Principle:Isaac sim IsaacGymEnvs Sim to Real Evaluation
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, Sim_to_Real |
| Last Updated | 2026-02-15 00:00 GMT |
Overview
Evaluation methodology for validating trained assembly policies using real-robot-compatible control signals and success metrics.
Description
Factory/IndustReal policies trained in simulation need evaluation with real-robot-compatible controllers before physical deployment. IndustReal provides generate_ctrl_signals() and simulate_and_refresh() methods that mirror the real robot's control interface. Evaluation tracks success rates (engagement detection, insertion depth) and validates that controller gains transfer to hardware.
The evaluation pipeline has three phases:
- Checkpoint loading: A trained policy checkpoint is loaded and the environment is configured for evaluation mode (deterministic actions, rendering enabled, metrics collection).
- Control signal generation: The
generate_ctrl_signals()method converts RL policy outputs to control signals using the same controller interface that would run on the real robot. This ensures the control commands are compatible with hardware. - Simulation and metrics:
simulate_and_refresh()steps the physics simulation and updates all tensor states. Success is measured by engagement detection (SDF queries showing insertion progress above threshold) and task completion within time limits.
Usage
Use this principle when evaluating assembly policies for sim-to-real transfer readiness. Specifically:
- Before real-robot deployment: Run evaluation in simulation with the hardware-compatible controller to verify success rates meet deployment thresholds.
- Controller gain tuning: Use evaluation mode to iterate on controller gains (stiffness, damping) that must work on both simulated and real robots.
- Policy comparison: Evaluate multiple training runs or hyperparameter configurations on standardized evaluation scenarios.
Theoretical Basis
Sim-to-real transfer validation requires ensuring three levels of compatibility:
# Sim-to-real compatibility levels
1. Control Interface Compatibility
- generate_ctrl_signals() in sim must produce the same command
format as the real robot controller
- Joint position targets (IK) or torques (impedance) must be
within hardware actuator limits
2. Dynamics Compatibility
- Controller gains (K_p, K_d) tuned in sim must produce stable
behavior on real hardware
- SAPU reward scaling during training helps avoid policies that
exploit simulation artifacts
3. Success Metric Compatibility
- Engagement detection (SDF-based) in sim must correlate with
physical engagement on hardware
- Insertion depth thresholds must account for real-world tolerances
The IndustReal approach uses the same control pipeline for training and evaluation, differing only in:
- Action determinism: Evaluation uses the policy mean (no exploration noise).
- Metrics collection: Success rates, insertion depths, and controller effort are logged.
- Rendering: Visualization is enabled for qualitative assessment.