Principle:OpenRLHF OpenRLHF Reward Shaping

Knowledge Sources	Proximal Policy Optimization Algorithms Training language models to follow instructions
Domains	Reinforcement_Learning
Last Updated	2026-02-07 00:00 GMT

Overview

A technique that combines environment reward with KL divergence penalty to create per-token reward signals for policy optimization.

Description

Reward Shaping transforms a sparse scalar reward (given at end of sequence) into a dense per-token reward signal by combining it with a KL divergence penalty at each token. The KL penalty discourages the policy from diverging too far from the reference model, while the environment reward (from a reward model or rule-based function) provides the task-specific signal.

Usage

Used in PPO and Math-GRPO workflows during experience generation. The shaped reward is used for advantage estimation and policy gradient computation.

Theoretical Basis

The shaped reward at each token position: $r_t=-\beta \cdot K{L}_t+\{\begin{array}{ll}R& \text{if }t=T_{eos}\\ 0& \text{otherwise}\end{array}$

where:

• $\beta$ (kl_coef) controls the KL penalty strength
• $K{L}_t$ is the per-token KL divergence estimate
• $R$ is the scalar reward placed at the EOS position
• $T_{eos}$ is the last non-padding token position

Related Pages

Implemented By

• Implementation:OpenRLHF_OpenRLHF_Compute_reward