Implementation:ARISE Initiative Robosuite CameraUtils
| Knowledge Sources | |
|---|---|
| Domains | Robotics, Computer Vision |
| Last Updated | 2026-02-15 07:00 GMT |
Overview
The camera_utils module provides utility functions and classes for camera operations in MuJoCo simulations, including intrinsic/extrinsic matrix computation, world-to-pixel projection, pixel-to-world back-projection, depth map conversion, and interactive camera manipulation during demonstration playback.
Description
This module contains two categories of functionality: stateless utility functions for camera math, and stateful classes for interactive camera control.
The utility functions include get_camera_intrinsic_matrix which computes the 3x3 camera matrix K from the MuJoCo camera's field of view and image dimensions; get_camera_extrinsic_matrix which returns the 4x4 camera-to-world homogeneous transformation with a correction for MuJoCo's camera axis convention; get_camera_transform_matrix which composes the intrinsic and extrinsic matrices to produce a full world-to-pixel projection matrix; get_real_depth_map which converts MuJoCo's normalized [0,1] depth values to actual metric distances using the near/far plane parameters; project_points_from_world_to_camera which batch-projects 3D world points to 2D pixel coordinates; and transform_from_pixels_to_world which back-projects pixel locations with depth values into 3D world coordinates using bilinear interpolation.
The CameraMover class provides runtime camera manipulation by attaching a camera to a mocap body in the MuJoCo scene. It modifies the simulation XML to replace a fixed camera with a mocap-mounted camera, enabling both translation (move_camera) and rotation (rotate_camera) during simulation. The DemoPlaybackCameraMover subclass extends this for demonstration replay, loading episode data from HDF5 files and supporting frame-by-frame playback with optional camera panning and domain randomization.
Usage
Use the stateless functions when you need to compute camera matrices or transform between world and pixel coordinate systems programmatically. Use CameraMover when you need to interactively position a camera at runtime. Use DemoPlaybackCameraMover when replaying recorded demonstrations with custom camera views.
Code Reference
Source Location
- Repository: ARISE_Initiative_Robosuite
- File: robosuite/utils/camera_utils.py
Signature
def get_camera_intrinsic_matrix(sim, camera_name, camera_height, camera_width) -> np.array: ...
def get_camera_extrinsic_matrix(sim, camera_name) -> np.array: ...
def get_camera_transform_matrix(sim, camera_name, camera_height, camera_width) -> np.array: ...
def get_camera_segmentation(sim, camera_name, camera_height, camera_width) -> np.array: ...
def get_real_depth_map(sim, depth_map) -> np.array: ...
def project_points_from_world_to_camera(points, world_to_camera_transform,
camera_height, camera_width) -> np.array: ...
def transform_from_pixels_to_world(pixels, depth_map,
camera_to_world_transform) -> np.array: ...
class CameraMover:
def __init__(self, env, camera="frontview", init_camera_pos=None, init_camera_quat=None): ...
def set_camera_pose(self, pos=None, quat=None): ...
def get_camera_pose(self) -> tuple: ...
def rotate_camera(self, point, axis, angle) -> tuple: ...
def move_camera(self, direction, scale) -> tuple: ...
Import
from robosuite.utils.camera_utils import (
get_camera_intrinsic_matrix,
get_camera_extrinsic_matrix,
get_camera_transform_matrix,
get_real_depth_map,
project_points_from_world_to_camera,
transform_from_pixels_to_world,
CameraMover,
)
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| sim | MjSim | Yes | Simulator instance with camera model data |
| camera_name | str | Yes | Name of the camera in the MuJoCo model |
| camera_height | int | Yes | Height of camera images in pixels |
| camera_width | int | Yes | Width of camera images in pixels |
| depth_map | np.array | Yes (for depth conversion) | Normalized [0,1] depth map from MuJoCo |
| points | np.array | Yes (for projection) | 3D world points of shape [..., 3] |
| world_to_camera_transform | np.array | Yes (for projection) | 4x4 transformation matrix |
Outputs
| Name | Type | Description |
|---|---|---|
| K (intrinsic) | np.array (3x3) | Camera intrinsic matrix |
| R (extrinsic) | np.array (4x4) | Camera extrinsic matrix (pose in world frame) |
| transform_matrix | np.array (4x4) | Full world-to-pixel projection matrix |
| depth_map (real) | np.array | Depth map with actual metric distances |
| pixels | np.array [..., 2] | Projected pixel coordinates (height, width) |
| points | np.array [..., 3] | 3D world coordinates from back-projection |
Usage Examples
import numpy as np
from robosuite.utils.camera_utils import (
get_camera_intrinsic_matrix,
get_camera_extrinsic_matrix,
get_camera_transform_matrix,
get_real_depth_map,
project_points_from_world_to_camera,
)
# Assuming `sim` is an active MjSim instance
camera_name = "agentview"
height, width = 256, 256
# Get camera matrices
K = get_camera_intrinsic_matrix(sim, camera_name, height, width)
R = get_camera_extrinsic_matrix(sim, camera_name)
T = get_camera_transform_matrix(sim, camera_name, height, width)
# Project a 3D world point to pixel coordinates
world_point = np.array([[0.5, 0.0, 0.8]]) # shape (1, 3)
pixel = project_points_from_world_to_camera(world_point, T, height, width)
print(f"Projected pixel: {pixel}")
# Convert MuJoCo depth map to real distances
rgb, depth = sim.render(width=width, height=height, camera_name=camera_name, depth=True)
real_depth = get_real_depth_map(sim, depth)