Implementation:Haosulab ManiSkill ObservationUtils

Knowledge Sources	Haosulab_ManiSkill
Domains	Robotics, Simulation, Observation Processing
Last Updated	2026-02-15 08:00 GMT

Overview

Concrete tool for converting raw sensor data into specific observation formats such as point clouds.

Description

The observations.py module provides functions that transform raw observation data (particularly camera sensor output) into structured formats suitable for agent consumption. The primary function is sensor_data_to_pointcloud().

sensor_data_to_pointcloud: Converts camera image data into world-space point cloud observations. For each camera sensor:

Extracts the position texture (x, y, z per pixel in OpenGL camera space) and segmentation data.
Converts depth from millimeters to meters.
Transforms points from camera space to world space using the cam2world_gl transformation matrix.
Produces an xyzw tensor where the 4th channel indicates whether the point is valid (non-background).
Optionally includes RGB and segmentation channels.
Merges point clouds from all cameras into a single observation under the "pointcloud" key.

The function modifies the observation dictionary in-place, moving processed camera data from "sensor_data" to "pointcloud".

Usage

Used internally by ManiSkill's observation pipeline when the observation mode is set to "pointcloud". Not typically called directly by users.

Code Reference

Source Location

Repository: Haosulab_ManiSkill
File: mani_skill/envs/utils/observations/observations.py

Signature

def sensor_data_to_pointcloud(
    observation: dict,
    sensors: dict[str, BaseSensor]
) -> dict: ...

Import

from mani_skill.envs.utils.observations.observations import sensor_data_to_pointcloud

I/O Contract

Inputs

Name	Type	Required	Description
observation	dict	Yes	Observation dictionary containing "sensor_data" and "sensor_param" keys
sensors	dict[str, BaseSensor]	Yes	Dictionary of sensor objects keyed by sensor UID

Outputs

Name	Type	Description
observation	dict	Modified observation dict with "pointcloud" key containing xyzw, rgb, segmentation tensors

Usage Examples

Basic Usage

from mani_skill.envs.utils.observations.observations import sensor_data_to_pointcloud

# Typically called within the observation pipeline
observation = sensor_data_to_pointcloud(observation, env.scene.sensors)
# observation["pointcloud"]["xyzw"] -> (B, num_points, 4)
# observation["pointcloud"]["rgb"]  -> (B, num_points, 3)

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment