Workflow:Volcengine Verl PPO Training With Reward Model
| Knowledge Sources | |
|---|---|
| Domains | LLMs, Reinforcement_Learning, RLHF, Post_Training |
| Last Updated | 2026-02-07 18:00 GMT |
Overview
End-to-end process for training large language models using Proximal Policy Optimization (PPO) with a learned reward model, implementing the full RLHF (Reinforcement Learning from Human Feedback) pipeline in verl.
Description
This workflow implements the classic RLHF training loop using PPO with an actor-critic architecture. Unlike GRPO, PPO requires both a policy network (actor) and a value network (critic) for Generalized Advantage Estimation (GAE). A separate learned reward model scores generated responses, providing the training signal. This workflow supports both colocated and split placement strategies for distributing actor, critic, reward model, and rollout workers across GPUs.
Usage
Execute this workflow when you need to align an LLM using human preference data or a trained reward model, particularly when verifiable rule-based rewards are not available. This is appropriate for tasks like general instruction following, helpfulness/harmlessness alignment (HH-RLHF), or any scenario where a discriminative or generative reward model provides the training signal.
Execution Steps
Step 1: Environment Setup
Install verl with training backend (FSDP or Megatron-LM), inference engine (vLLM or SGLang), and reward model dependencies. Initialize a Ray cluster with sufficient GPU resources to host actor, critic, reward model, and rollout workers simultaneously.
Key considerations:
- PPO requires more GPU memory than GRPO due to the additional critic and reward model
- Split placement can distribute models across separate GPU pools for better memory efficiency
- Colocated placement shares GPU resources between workers using time-multiplexing
Step 2: Data Preparation
Prepare training data in verl's parquet format with the reward model style set to "model" instead of "rule". For RLHF datasets, this involves converting human preference data (chosen/rejected pairs) into the standard prompt format. Three data splits may be needed: SFT data for initial fine-tuning, RM data for reward model training, and RL data for policy optimization.
Key considerations:
- Set reward_model.style to "model" in the data schema
- The prompt field uses the standard chat message format
- For HH-RLHF style data, chosen/rejected pairs are used to train the reward model separately
- The RL split only needs prompts (responses are generated during rollout)
Step 3: Model Initialization
Load the actor model, critic model, reference model, and reward model. The actor and reference start from the same pretrained checkpoint. The critic is typically initialized from the same architecture. The reward model is a separately trained model that outputs scalar reward scores.
Key considerations:
- Actor and reference models share the same architecture and initial weights
- The critic can use the same or different architecture from the actor
- The reward model (e.g., FsfairX-LLaMA3-RM-v0.1) runs in its own worker group
- Reference model weights are frozen and used only for KL divergence computation
Step 4: Rollout Generation
Generate response completions for training prompts using the current actor policy via the rollout engine (vLLM or SGLang). Unlike GRPO, PPO typically generates fewer samples per prompt since the critic provides per-token value estimates rather than relying on group comparisons.
Key considerations:
- Tensor parallel configuration for rollout is independent of training configuration
- The rollout engine shares weights with the actor through verl's weight synchronization mechanism
- Log probabilities from the rollout are stored for computing the importance sampling ratio
Step 5: Reward Model Scoring
Pass generated responses through the learned reward model to obtain scalar reward scores. The reward model processes the full prompt-response pair and outputs a score indicating response quality according to human preferences.
Key considerations:
- Reward model runs on dedicated workers with its own tensor parallelism configuration
- The reward loop interface supports both colocated and standalone reward computation
- Reward scores can be augmented with KL penalty terms computed against the reference model
- GPU memory utilization for the reward model should be configured separately
Step 6: Advantage Estimation with GAE
Compute per-token advantages using Generalized Advantage Estimation (GAE), which combines the reward signal with critic value predictions to produce low-variance advantage estimates. The critic is trained to predict the value function while the actor is updated using the clipped surrogate objective.
Key considerations:
- GAE lambda (typically 0.95-1.0) controls the bias-variance tradeoff
- The critic is updated for multiple epochs on the same trajectory data
- Both actor and critic use separate learning rates (critic typically 10x higher)
- Dual-clip PPO can be enabled for additional stability
Step 7: Evaluation and Checkpointing
Evaluate the policy on held-out test prompts and save checkpoints of actor, critic, and optimizer states. Monitor training stability through reward trends, KL divergence, and policy entropy metrics.
Key considerations:
- Both actor and critic checkpoints need to be saved for training resumption
- The reward model checkpoint is static and does not need saving
- Early divergence in KL or reward collapse indicates training instability
- Sequence balancing can improve GPU utilization for variable-length responses