Principle:Haosulab ManiSkill Real Robot Deployment
| Field | Value |
|---|---|
| Principle Name | Real Robot Deployment |
| Domain | Robotics |
| Overview | Deploying trained policies to physical robots through hardware abstraction agents |
| Date | 2026-02-15 |
| Repository | Haosulab/ManiSkill |
Overview
The Real Robot Deployment principle describes how ManiSkill provides an abstract interface for controlling physical robots, capturing sensor data, and reading proprioceptive state. This hardware abstraction layer allows the sim2real bridge (see Principle:Haosulab_ManiSkill_Sim2Real_Bridge) to communicate with different physical robot platforms through a uniform API, decoupling the deployment logic from specific robot hardware.
Description
The deployment architecture is built around two key abstractions:
BaseRealAgent
An abstract base class that defines the interface every real robot driver must implement:
- Lifecycle management:
start()to initialize hardware (power motors, connect cameras) andstop()to safely shut down. - Motor control:
set_target_qpos(qpos)to command joint positions andset_target_qvel(qvel)to command joint velocities, mirroring the simulation's drive target interface. - Safe reset:
reset(qpos)to move the robot to a target configuration at a controlled speed, avoiding jerky motions that could damage hardware or the environment. - Sensor capture:
capture_sensor_data(sensor_names)for asynchronous sensor reading andget_sensor_data(sensor_names)for retrieving the captured data in a structured format compatible with the simulation's sensor output. - Proprioception:
get_qpos()andget_qvel()to read current joint positions and velocities, plusget_proprioception()which combines them with controller state.
The BaseRealAgent also provides a RealRobot inner class that mimics the simulation robot's interface (.qpos, .qvel properties), enabling the sim2real bridge to access robot state through the same attribute paths.
Concrete Implementations (e.g., LeRobotRealAgent)
Concrete agent classes implement the abstract interface for specific hardware platforms. The LeRobotRealAgent connects to robots supported by the LeRobot library (from Hugging Face), which includes low-cost manipulators like the SO-100. Key implementation details:
- Motor communication: Uses LeRobot's
Robotclass for reading joint positions and sending joint commands. - Unit conversion: Converts between radians (ManiSkill's internal representation) and degrees (LeRobot's motor interface).
- Smooth reset: Implements gradual position interpolation during reset, moving at a maximum rate of 0.025 radians per step at 30 Hz to prevent sudden motions.
- QPos caching: Optionally caches joint position readings to reduce latency (reading from physical motors can take 5-40ms).
- Camera integration: Uses LeRobot's camera abstraction with asynchronous reading for low-latency visual observations.
Usage
from lerobot.common.robots.robot import Robot
from mani_skill.agents.robots.lerobot.manipulator import LeRobotRealAgent
# Create and configure the LeRobot robot instance
robot = Robot(name="so100_follower", config=my_config)
agent = LeRobotRealAgent(robot=robot, use_cached_qpos=True)
agent.start()
# Use with Sim2RealEnv for policy deployment
sim2real_env = Sim2RealEnv(sim_env=sim_env, agent=agent)
obs, info = sim2real_env.reset()
# ... run policy ...
sim2real_env.close()
Theoretical Basis
- Hardware abstraction: The
BaseRealAgentinterface follows the adapter pattern, providing a uniform API over diverse hardware backends. This is a standard robotics software engineering practice (cf. ROS hardware interfaces) that enables code reuse and portability.
- Real-time control: Physical robots require deterministic, timely control commands. The agent interface separates the what (desired joint positions) from the how (motor communication protocol), allowing each implementation to handle timing constraints appropriately.
- Sensor fusion: By providing a structured sensor data format that mirrors the simulation's output, the agent enables the sim2real bridge to process real and simulated observations identically, facilitating zero-shot transfer.
- Safe motion: The gradual reset mechanism (position interpolation with bounded step size) is a critical safety feature for real robots. Sudden large joint motions can damage the robot, its environment, or pose safety risks to humans. The reset function ensures smooth, bounded velocity motions.
- Latency management: QPos caching is a practical optimization that trades off freshness of state information for higher control frequencies. This tradeoff is acceptable when the robot moves slowly relative to the control frequency, which is typical for manipulation tasks.