Environment:OpenRLHF OpenRLHF CUDA GPU Environment

Knowledge Sources	OpenRLHF NVIDIA CUDA Toolkit
Domains	Infrastructure, Deep_Learning, Distributed_Training
Last Updated	2026-02-07 10:00 GMT

Overview

Linux environment with NVIDIA CUDA GPUs, Python >= 3.10, PyTorch with distributed support, and bf16/fp16 mixed precision.

Description

This environment defines the core hardware and software stack required by all OpenRLHF training workflows. OpenRLHF is classified under Environment :: GPU :: NVIDIA CUDA and requires NVIDIA GPUs for CUDA-accelerated distributed training. The framework uses PyTorch's distributed backend with NCCL for inter-GPU communication, and supports bf16 (default) or fp16 mixed precision training. Multi-GPU setups are required for distributed training with DeepSpeed ZeRO stages 0-3, tensor parallelism, and ring attention.

Usage

Use this environment for all OpenRLHF training and inference workflows. Every training script (SFT, DPO, PPO, RM, KD, KTO, PRM) and every inference tool (batch inference, interactive chat, reward model serving) requires CUDA GPU access. This is the foundational prerequisite that all other environments build upon.

System Requirements

Category	Requirement	Notes
OS	Linux (Ubuntu recommended)	`manylinux1_x86_64` wheel target in setup.py
Hardware	NVIDIA CUDA GPU	Multi-GPU recommended for distributed training
GPU Memory	16GB+ VRAM recommended	Depends on model size; 7B models need ~16GB with bf16
CPU	Multi-core with sufficient RAM	CPU offloading requires adequate system RAM
Network	High-bandwidth inter-node	Required for multi-node distributed training via NCCL

Dependencies

System Packages

NVIDIA CUDA Toolkit (compatible with PyTorch version)
NCCL (for distributed GPU communication)

Python Packages

`python` >= 3.10
`torch` (with CUDA support)
`transformers` == 4.57.6
`deepspeed` == 0.18.5
`accelerate`
`peft`
`datasets`
`bitsandbytes` (for 4-bit quantization)
`pynvml` >= 12.0.0
`optree` >= 0.15.0
`torchdata`
`torchmetrics`
`einops`
`packaging`
`loralib`
`optimum`
`tqdm`

Credentials

The following environment variables may be required depending on the workflow:

`LOCAL_RANK`: Set automatically by distributed launcher; GPU device assignment
`MASTER_ADDR`: Master node address for distributed training
`MASTER_PORT`: Master node port for distributed training
`WORLD_SIZE`: Total number of distributed processes
`RANK`: Global rank of the current process
`NCCL_CUMEM_ENABLE`: NCCL communication memory flag (set to "0" for PPO actor)
`TOKENIZERS_PARALLELISM`: Set to "true" by Ray PPO launcher
`NCCL_DEBUG`: Set to "WARN" by Ray PPO launcher

Quick Install

# Install OpenRLHF with all core dependencies
pip install openrlhf

# Or install from requirements.txt
pip install torch transformers==4.57.6 deepspeed==0.18.5 accelerate peft datasets \
    bitsandbytes pynvml>=12.0.0 optree>=0.15.0 torchdata torchmetrics einops \
    packaging loralib optimum tqdm wandb tensorboard

Code Evidence

Python version requirement from `setup.py:82`:

python_requires=">=3.10",

CUDA device selection from `openrlhf/utils/deepspeed/deepspeed.py:92-94`:

local_rank = int(os.environ.get("LOCAL_RANK", "-1"))
if local_rank != -1:
    torch.cuda.set_device(local_rank)

CUDA device mesh initialization from `openrlhf/utils/deepspeed/deepspeed.py:100-104`:

self.world_size = dist.get_world_size()
dp_size = self.world_size // self.ring_attn_size // self.ds_tensor_parallel_size
self.ds_device_mesh = init_device_mesh(
    "cuda", (dp_size, self.ring_attn_size, self.ds_tensor_parallel_size), mesh_dim_names=("dp", "sp", "tp")
)

Parameter dtype validation from `openrlhf/utils/utils.py:8-22`:

def convert_to_torch_dtype(param_dtype: str) -> torch.dtype:
    if param_dtype == "bf16":
        return torch.bfloat16
    elif param_dtype == "fp16":
        return torch.float16
    else:
        raise ValueError(f"Invalid param_dtype: {param_dtype}")

BF16 requirement for tensor parallelism from `openrlhf/utils/deepspeed/deepspeed.py:74`:

assert self.param_dtype == "bf16", "BF16 is required for tensor parallel training"

GPU environment classifier from `setup.py:87`:

"Environment :: GPU :: NVIDIA CUDA",

Common Errors

Error Message	Cause	Solution
`Invalid param_dtype: {param_dtype}`	Unsupported dtype string passed	Use "bf16" (recommended) or "fp16"
`BF16 is required for tensor parallel training`	fp16 used with tensor parallelism	Switch to `--param_dtype bf16`
`CUDA out of memory`	Insufficient GPU VRAM	Enable gradient checkpointing, use adam_offload, reduce batch size
NCCL timeout	Network issues in distributed setup	Check MASTER_ADDR/MASTER_PORT; increase timeout

Compatibility Notes

bf16 vs fp16: bf16 is the default and recommended dtype. Tensor parallel training requires bf16. 4-bit quantization (BitsAndBytes) also requires bf16.
Multi-GPU: World size must be divisible by `ring_attn_size * ds_tensor_parallel_size` for device mesh creation.
Linux only: The build system targets `manylinux1_x86_64`; Windows/macOS not officially supported.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment