Principle:Hiyouga LLaMA Factory Proximal Policy Optimization

Knowledge Sources	Hiyouga_LLaMA_Factory Proximal Policy Optimization Algorithms Training language models to follow instructions with human feedback
Domains	Reinforcement Learning, Natural Language Processing, Language Model Alignment
Last Updated	2026-02-06 19:00 GMT

Overview

A reinforcement learning from human feedback (RLHF) training paradigm that uses a learned reward model and the proximal policy optimization algorithm to align language model outputs with human preferences.

Description

Proximal Policy Optimization (PPO) applied to language model alignment is the canonical RLHF approach popularized by OpenAI's InstructGPT (Ouyang et al., 2022). The method involves three models working together: a policy model that generates responses, a reward model that scores those responses, and a reference model that provides a KL-divergence anchor to prevent the policy from diverging too far from the original pretrained distribution.

The PPO-based RLHF pipeline operates in an online fashion:

1. The policy model generates responses to prompts from the training dataset.
2. The reward model scores each generated response.
3. The PPO algorithm updates the policy to maximize the reward while staying close to the reference model.

This approach is more complex than offline methods like DPO but offers several advantages:

• Online learning: The model learns from its own generations rather than a fixed dataset, enabling exploration.
• Flexible reward signal: The reward model can encode complex, non-decomposable preferences.
• Proven effectiveness: PPO-based RLHF has been demonstrated at scale in production systems (ChatGPT, Claude).
• Value head: A learned value function estimates expected future rewards, reducing variance in policy gradient estimates.

The main challenges include training instability, computational cost (multiple models in memory), and sensitivity to hyperparameters.

Usage

Use PPO-based RLHF when you want to:

• Align a language model using a separately trained reward model.
• Perform online exploration where the model learns from its own generated responses.
• Apply sophisticated reward shaping that goes beyond pairwise preferences.
• Follow the classic InstructGPT/ChatGPT alignment pipeline.

PPO is most appropriate when you have a reliable reward model, sufficient compute for multi-model training, and the need for online policy improvement.

Theoretical Basis

RLHF Objective

The RLHF objective maximizes expected reward while constraining the KL divergence from the reference policy:

$\underset{{\pi }_{\theta }}{\max }{\mathbb{𝔼}}_{x\sim {𝒟},y\sim {\pi }_{\theta }(\cdot \mid x)}[R(x,y)-\beta \text{KL}({\pi }_{\theta }(\cdot \mid x)\Vert {\pi }_{\text{ref}}(\cdot \mid x))]$

where $R(x,y)$ is the reward model score, ${\pi }_{\theta }$ is the policy, ${\pi }_{\text{ref}}$ is the reference model, and $\beta$ is the KL penalty coefficient.

PPO Clipped Objective

PPO optimizes a clipped surrogate objective to ensure stable updates:

${{ℒ}}^{\text{CLIP}}(\theta )={\mathbb{𝔼}}_t[\min (r_t(\theta ){\hat{A}}_t,\text{clip}(r_t(\theta ),1-ϵ,1+ϵ){\hat{A}}_t)]$

where $r_t(\theta )=\frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{{\theta }_{\text{old}}}(a_t\mid s_t)}$ is the probability ratio, ${\hat{A}}_t$ is the estimated advantage, and $ϵ$ is the clipping parameter. The clipping prevents excessively large policy updates that could destabilize training.

Generalized Advantage Estimation

The advantage function is estimated using Generalized Advantage Estimation (GAE):

${\hat{A}}_t=\sum _{l=0}^{T-t}(\gamma \lambda {)}^l{\delta }_{t+l}$

where ${\delta }_t=r_t+\gamma V(s_{t+1})-V(s_t)$ is the TD residual, $\gamma$ is the discount factor, and $\lambda$ is the GAE parameter. In the language model setting, the value function $V(s_t)$ is provided by a value head -- a linear layer appended to the language model that estimates the expected cumulative reward from each token position.

Reward Computation

The reward for a generated response is obtained from:

• Full reward model: A separate model with a value head that outputs a scalar reward at the last token position.
• LoRA reward model: The same base model with a separate LoRA adapter switched in to provide reward scores.
• API reward model: An external reward service that scores generated text.

Rewards can optionally be whitened (normalized to zero mean and unit variance) or score-normalized to stabilize training.

Training Loop

Each PPO step involves:

1. Generation phase: The policy generates responses for a buffer of prompts.
2. Reward phase: The reward model scores each generated response.
3. Optimization phase: Multiple PPO epochs update the policy on the collected buffer, using the clipped objective and value function loss.

The buffer size, mini-batch size, and number of PPO epochs per buffer are key hyperparameters that control the trade-off between sample efficiency and computational cost.

Related Pages

• Implementation:Hiyouga_LLaMA_Factory_PPO_Trainer
• Implementation:Hiyouga_LLaMA_Factory_PPO_Workflow