Principle:Hpcaitech ColossalAI GRPO Policy Loss

Knowledge Sources	ColossalAI DeepSeekMath: Pushing the Limits of Mathematical Reasoning Proximal Policy Optimization Algorithms
Domains	Reinforcement_Learning, Optimization
Last Updated	2026-02-09 00:00 GMT

Overview

A policy gradient loss function that uses PPO-style clipping with group-relative advantages and a KL divergence penalty for stable reinforcement learning.

Description

The GRPO Policy Loss combines PPO's clipped surrogate objective with a per-token KL divergence penalty against a reference model. The advantages are computed group-relatively (normalized within each prompt's group of generations), eliminating the need for a learned value function. The loss supports both sample-level and token-level aggregation.

Usage

Use this loss function inside the GRPO consumer's training step to update the policy model based on collected experiences.

Theoretical Basis

The clipped policy loss with KL penalty:

$ℒ = - \frac{1}{N} \sum_{i} \min (r_{i} (θ) A_{i}, clip (r_{i} (θ), 1 - ϵ, 1 + ϵ) A_{i}) + β \cdot K L_{t o k e n}$

Where $r_{i} (θ) = \frac{π_{θ} (a_{i} | s_{i})}{π_{o l d} (a_{i} | s_{i})}$ is the importance sampling ratio.

Related Pages

Implemented By

Implementation:Hpcaitech_ColossalAI_PolicyLoss

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment