Principle:CarperAI Trlx Reward Function Design

Knowledge Sources	Learning to Summarize from Human Feedback Training language models to follow instructions with human feedback CarperAI trlx
Domains	Reinforcement_Learning, Reward_Modeling, NLP
Last Updated	2026-02-07 16:00 GMT

Overview

A design principle for creating reward functions that score language model generations during online reinforcement learning training.

Description

In RLHF, the reward function is the critical signal that guides the language model toward desired behavior. During PPO training, the model generates text from prompts, and a reward function scores each generation. These scores drive the policy gradient updates. Reward functions can be rule-based (e.g., sentiment classifiers), model-based (trained reward models), or hybrid approaches.

The reward function must satisfy a specific interface: it receives generated text samples and optionally the original prompts and decoded outputs, and returns a list of scalar rewards. The quality and calibration of the reward function directly determines the quality of the RL-trained model.

Usage

Design a reward function when setting up online PPO training with trlx.train(). The reward function is passed as the reward_fn argument and is called during each rollout to score generated samples. Choose this approach when you have a computable measure of text quality (sentiment, toxicity, factual accuracy, format compliance).

Theoretical Basis

The reward function maps generated text to scalar values:

$r:{𝒴}\to \mathbb{ℝ}$

In PPO, the reward signal is combined with a KL penalty to prevent reward hacking:

$R(x,y)=r(y)-\beta \cdot KL({\pi }_{\theta }(y|x)\Vert {\pi }_{ref}(y|x))$

Design considerations:

• Scale: Rewards should be bounded and normalized to prevent instability
• Density: Per-sample scalar rewards (not sparse episode-level rewards)
• Calibration: Higher values should correspond to genuinely better text
• Speed: The reward function is called at every rollout step, so inference speed matters

Common reward function patterns:

• Rule-based: Sentiment classifiers, regex matching, length constraints
• Model-based: Trained reward models that predict human preferences
• Delta rewards: Compute reward difference between generated and reference output

Related Pages

Implemented By

• Implementation:CarperAI_Trlx_Reward_Function_Interface

Uses Heuristic

• Heuristic:CarperAI_Trlx_Delta_Rewards