Principle:Alibaba ROLL Advantage Estimation with KL Penalty

Knowledge Sources	PPO GRPO GAE Alibaba ROLL
Domains	Reinforcement_Learning, Optimization
Last Updated	2026-02-07 20:00 GMT

Overview

An advantage estimation principle that combines multiple estimation algorithms (GAE, GRPO, Reinforce++) with token-level KL divergence penalties to produce stable policy gradient signals.

Description

Advantage estimation transforms raw reward signals into normalized, variance-reduced training signals for policy gradient methods. This principle covers three interconnected operations:

1. Reward post-processing: Normalizing and clipping response-level rewards using running statistics
2. Token-level KL penalty: Adding a per-token KL divergence penalty between the current policy and a reference policy to prevent reward hacking
3. Advantage computation: Computing per-token advantages using one of several estimators (GAE, GRPO, Reinforce++)

The KL penalty is particularly important in RLHF/RLVR as it prevents the policy from diverging too far from the reference model, which would lead to degenerate outputs that exploit reward model weaknesses.

Usage

Use this principle after reward computation and before policy optimization. The choice of advantage estimator affects training dynamics:

• GRPO: Group-relative, no value function needed, normalizes within a group of samples per prompt
• Reinforce++: Standard baseline subtraction
• GAE: Generalized Advantage Estimation with value function, requires a critic network

Theoretical Basis

KL-Penalized Reward

$r_t^{KL}=r_t-\beta \cdot KL[{\pi }_{\theta }(a_t|s_t)\Vert {\pi }_{ref}(a_t|s_t)]$

Where $\beta$ is an adaptive coefficient updated based on a target KL budget.

Advantage Estimators

GAE: ${\hat{A}}_t^{GAE}=\sum _{l=0}^{\mathrm{\infty }}(\gamma \lambda {)}^l{\delta }_{t+l},{\delta }_t=r_t+\gamma V(s_{t+1})-V(s_t)$

GRPO (Group Relative): ${\hat{A}}_i=\frac{r_i-{\mu }_G}{{\sigma }_G}$

Where ${\mu }_G$ and ${\sigma }_G$ are the mean and std of rewards within the group.

Reinforce++: ${\hat{A}}_t=R_t-b(s_t)$

Where $b(s_t)$ is the running mean baseline.

Related Pages

Implemented By

• Implementation:Alibaba_ROLL_Compute_Advantage

Related Heuristics

The following heuristics inform this principle:

• Heuristic:Alibaba_ROLL_PPO_Clipping_Defaults
• Heuristic:Alibaba_ROLL_Numerical_Stability_Epsilon
• Heuristic:Alibaba_ROLL_KL_Coefficient_Tuning
• Heuristic:Alibaba_ROLL_Reward_Clipping_Normalization