Workflow:Haosulab ManiSkill Sim2Real Deployment
| Knowledge Sources | |
|---|---|
| Domains | Sim2Real_Transfer, Robotics_Deployment, Domain_Randomization |
| Last Updated | 2026-02-15 12:00 GMT |
Overview
End-to-end process for deploying policies trained in ManiSkill simulation to physical robots using the Sim2RealEnv interface.
Description
This workflow covers the full pipeline from building a simulation environment that matches a real-world setup to deploying a trained policy on physical hardware. ManiSkill provides the Sim2RealEnv class that creates a unified Gymnasium interface between simulation and reality, automatically aligning action spaces, observation spaces, and control frequencies. The process includes building a digital twin simulation environment, applying domain randomization to bridge the sim-to-real gap, training a robust policy, and deploying through the Sim2RealEnv bridge with sensor data preprocessing.
Usage
Execute this workflow when you have a trained simulation policy and want to deploy it on a physical robot. This is appropriate when you have access to a supported real robot (e.g., SO100, Koch, or any robot with a LeRobot-compatible driver), a calibrated workspace matching your simulation scene, and cameras positioned to provide the expected observations.
Execution Steps
Step 1: Digital Twin Environment Construction
Build a simulation environment that closely mirrors the physical workspace. This involves matching the real robot model (URDF), scene geometry (table dimensions, camera positions), and object properties. Use ManiSkill's scene builders and asset loading to recreate the physical setup, paying attention to coordinate frame alignment and scale accuracy.
Key considerations:
- Camera intrinsics and extrinsics must match the real sensor configuration
- Table height, workspace boundaries, and object dimensions should be measured from the real setup
- Use the same robot URDF that describes the physical robot's kinematics
- The digital twin tasks inherit from BaseDigitalTwinEnv for built-in sim2real support
Step 2: Domain Randomization
Apply domain randomization to the simulation environment to train policies that are robust to sim-to-real distribution shifts. Randomize visual properties (lighting, textures, camera poses), physical properties (friction, mass), and initial conditions (object poses, robot starting configuration). ManiSkill provides built-in randomization utilities for camera poses, object placements, and material properties.
Key considerations:
- Randomize camera poses within a small range to handle calibration errors
- Vary object colors, textures, and lighting conditions for visual robustness
- Randomize dynamics parameters (friction coefficients, damping) within reasonable ranges
- Apply randomization during reconfiguration for heterogeneous parallel environments
Step 3: Policy Training with Sim2Real Focus
Train the policy using RL or imitation learning with domain randomization active. The training process follows the standard ManiSkill training pipeline but with the randomized environment. Focus on observation representations that transfer well (e.g., proprioception + RGB rather than privileged state) and control modes that match the real robot's interface.
Key considerations:
- Avoid relying on privileged state information (object poses) that is unavailable on real hardware
- Use sensor-based observations (RGBD) that can be captured by real cameras
- Match the control mode to what the real robot driver supports
- Train with the same control frequency that will be used on the real robot
Step 4: Sim2RealEnv Bridge Setup
Configure the Sim2RealEnv class to bridge the simulation and real environments. This involves instantiating the BaseRealAgent for the physical robot, setting up sensor data preprocessing (center crop and resize to match simulation image dimensions), and configuring the reset procedure for the real environment.
Key considerations:
- The Sim2RealEnv automatically aligns action and observation spaces between sim and real
- Sensor preprocessing handles resolution differences between sim cameras and real cameras
- A custom reset function can automate real-world episode resets
- The control frequency is matched between simulation and real robot
Step 5: Real Robot Deployment and Evaluation
Deploy the trained policy on the physical robot through the Sim2RealEnv interface. The policy receives real sensor observations (preprocessed to match simulation format) and outputs actions that are sent to the real robot. Run evaluation episodes to measure real-world task success rate and identify any remaining sim-to-real gaps.
Key considerations:
- Start with slow execution speed and gradually increase to the trained control frequency
- Monitor the robot during initial deployments for unexpected behaviors
- Compare real-world success rate to simulation evaluation metrics
- Iterate on domain randomization parameters if the sim-to-real gap is too large