Principle:CARLA simulator Carla Sensor Data Processing
| Knowledge Sources | |
|---|---|
| Domains | Simulation, Perception, Sensor_Fusion |
| Last Updated | 2026-02-15 00:00 GMT |
Overview
Sensor data processing encompasses the conversion, transformation, and serialization of raw sensor observations into usable formats for storage, visualization, and downstream machine learning or perception pipelines.
Description
Each CARLA sensor modality produces data in a specific raw format that must be processed before it can be saved to disk or consumed by downstream systems. The processing requirements vary by sensor type:
Camera Sensors (RGB, Depth, Semantic Segmentation):
- Raw data is a flat BGRA byte array with shape (height * width * 4).
- Image.save_to_disk() saves directly to common image formats (PNG, JPEG) with optional color conversion.
- Data can be converted to a numpy array via np.frombuffer() for in-memory processing.
- Color converters transform raw sensor output into human-readable or algorithm-friendly formats:
- Raw: Unmodified sensor output (default).
- Depth: Converts encoded depth buffer to grayscale depth image.
- LogarithmicDepth: Applies logarithmic scaling for better visualization of depth range.
- CityScapesPalette: Maps semantic segmentation class IDs to the CityScapes color palette.
LiDAR Sensor:
- Raw data is a structured array of 3D points with intensity values.
- Accessible as a numpy array with shape (N, 4) where columns are (x, y, z, intensity).
- Can be saved as PLY point cloud files via save_to_disk().
- Horizontal angle information is available for organizing points into scan lines.
Radar Sensor:
- Raw data is an array of radar detections, each containing: velocity, azimuth, altitude, and depth.
- Accessible via iteration over the RadarMeasurement object.
GNSS Sensor:
- Provides latitude, longitude, and altitude as floating-point attributes.
IMU Sensor:
- Provides accelerometer (Vector3D), gyroscope (Vector3D), and compass (float, in radians) readings.
Usage
Data processing is performed inside the collection loop, after retrieving synchronized data from sensor queues. The processing step converts raw sensor data into the desired output format (images, point clouds, structured logs) and writes them to disk or feeds them to real-time processing pipelines.
Theoretical Basis
Image Color Space Conversion: CARLA cameras render in BGRA (Blue, Green, Red, Alpha) byte order, following the convention of Unreal Engine's rendering pipeline. For machine learning applications, conversion to RGB or BGR (OpenCV convention) is typically required. The conversion is a simple channel reorder operation on the numpy array.
Depth Encoding: CARLA's depth camera encodes depth information across the RGB channels of an image using the formula: depth_meters = 1000 * (R + G * 256 + B * 65536) / (256^3 - 1). This provides millimeter-level precision up to 1 kilometer. The Depth color converter linearizes this into a grayscale image, while LogarithmicDepth applies a logarithmic transformation: output = log(depth) / log(max_depth), which provides better perceptual contrast across the depth range.
Semantic Segmentation Labels: The semantic segmentation camera assigns each pixel a class ID corresponding to a predefined set of categories (road, sidewalk, vehicle, pedestrian, vegetation, building, etc.). The CityScapesPalette converter maps these IDs to the CityScapes dataset color scheme, enabling direct visual comparison with real-world semantic segmentation benchmarks.
Point Cloud Representation: LiDAR data is represented as an unordered set of 3D points in the sensor's local coordinate frame. Each point carries an intensity value (0.0 to 1.0) representing the reflectivity of the surface that produced the return. The PLY (Polygon File Format) is used for disk storage, which supports both ASCII and binary encoding of point cloud data.
Data Serialization Formats: Choosing the right output format involves trade-offs between file size, read speed, and compatibility:
- PNG: Lossless compression, widely compatible, suitable for images and depth maps.
- JPEG: Lossy compression, smaller files, suitable for RGB images when exact pixel values are not critical.
- PLY: Standard point cloud format, compatible with Open3D, PCL, and other 3D processing libraries.
- NumPy (.npy): Fast binary serialization for programmatic access, preserving exact floating-point values.