Implementation:Haosulab ManiSkill MP Parallel Pattern

Field	Value
Implementation Name	MP Parallel Pattern
Type	Pattern Doc
Domain	Motion_Planning
Source File	`mani_skill/examples/motionplanning/panda/run.py` (L80-140)
Date	2026-02-15
Repository	Haosulab/ManiSkill

Overview

The MP Parallel Pattern implements multi-process parallelization for motion planning trajectory generation in ManiSkill. It uses Python's multiprocessing module to spawn independent worker processes, each generating a partition of the total requested trajectories, and then merges the results into a single dataset.

Description

The parallelization is implemented in the main() function of the runner script. When num_procs > 1, the function:

Divides num_traj evenly across num_procs processes.
Computes non-overlapping seed ranges for each process.
Creates a multiprocessing.Pool and dispatches work using pool.starmap.
After all processes complete, merges the per-process HDF5 files using merge_trajectories.
Cleans up by deleting the temporary per-process .h5 and .json files.

Each worker process receives a deep copy of the argument namespace, its process ID, and its starting seed. This ensures complete independence between workers.

Usage

# Generate 100 trajectories using 8 parallel processes
python -m mani_skill.examples.motionplanning.panda.run \
    -e StackCube-v1 -n 100 --num-procs 8 --record-dir demos

Code Reference

Parallel Dispatch (L127-147)

def main(args):
    if args.num_procs > 1 and args.num_procs < args.num_traj:
        if args.num_traj < args.num_procs:
            raise ValueError(
                "Number of trajectories should be greater than or equal to number of processes"
            )
        args.num_traj = args.num_traj // args.num_procs
        seeds = [*range(0, args.num_procs * args.num_traj, args.num_traj)]
        pool = mp.Pool(args.num_procs)
        proc_args = [(deepcopy(args), i, seeds[i]) for i in range(args.num_procs)]
        res = pool.starmap(_main, proc_args)
        pool.close()
        # Merge trajectory files
        output_path = res[0][: -len("0.h5")] + "h5"
        merge_trajectories(output_path, res)
        for h5_path in res:
            tqdm.write(f"Remove {h5_path}")
            os.remove(h5_path)
            json_path = h5_path.replace(".h5", ".json")
            tqdm.write(f"Remove {json_path}")
            os.remove(json_path)
    else:
        _main(args)

Process Entry Point Setup (L149-153)

if __name__ == "__main__":
    mp.set_start_method("spawn")
    main(parse_args())

The spawn start method is used instead of fork to avoid issues with CUDA context inheritance and to ensure clean process initialization.

Per-Process File Naming (L64-65)

if args.num_procs > 1:
    new_traj_name = new_traj_name + "." + str(proc_id)

This produces files like 20260215_120000.0.h5, 20260215_120000.1.h5, etc.

I/O Contract

Direction	Data	Format
Input	`args.num_procs`	`int` (number of parallel processes)
Input	`args.num_traj`	`int` (total trajectories to generate)
Intermediate	Per-process trajectory files	`{traj_name}.{proc_id}.h5` + `.json`
Output	Merged trajectory file	`{traj_name}.h5` + `.json`

Seed Assignment

Process ID	Start Seed	Num Trajectories
0	0	`num_traj // num_procs`
1	`num_traj // num_procs`	`num_traj // num_procs`
2	`2 * (num_traj // num_procs)`	`num_traj // num_procs`
...	...	...

For example, with num_traj=100 and num_procs=4, each process generates 25 trajectories with seeds [0..24], [25..49], [50..74], [75..99].

Usage Examples

# The parallel pattern is invoked via the CLI:
# python -m mani_skill.examples.motionplanning.panda.run -e PickCube-v1 -n 200 --num-procs 16

# Equivalent programmatic invocation:
import multiprocessing as mp
from mani_skill.examples.motionplanning.panda.run import main, parse_args

mp.set_start_method("spawn")
import sys
sys.argv = ["run.py", "-e", "PickCube-v1", "-n", "200", "--num-procs", "16"]
args = parse_args()
main(args)
# Result: a single merged .h5 file in demos/PickCube-v1/motionplanning/

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment