Principle:LLMBook zh LLMBook zh github io Reward Modeling

Knowledge Sources	Training language models to follow instructions with human feedback Learning to summarize from human feedback LLMBook-zh
Domains	NLP, Alignment, Reinforcement_Learning
Last Updated	2026-02-08 00:00 GMT

Overview

A technique that trains a scalar reward model on human preference data to score language model outputs for reinforcement learning alignment.

Description

Reward Modeling trains a model to predict human preferences between pairs of model outputs. Given a prompt and two responses (one preferred, one rejected), the reward model learns to assign higher scalar rewards to preferred responses. The trained reward model then provides the reward signal for reinforcement learning algorithms like PPO.

The approach uses contrastive learning: the model computes rewards for both responses and optimizes a binary cross-entropy loss on the reward difference. An optional language modeling loss serves as a regularization term to prevent the reward model from forgetting language understanding.

Usage

Use this principle when implementing the RLHF (Reinforcement Learning from Human Feedback) pipeline. The reward model is trained before the RL fine-tuning stage and is used to score model outputs during PPO/GRPO training.

Theoretical Basis

Given a preferred response $y_w$ and a rejected response $y_l$:

1. Compute scalar rewards: $r_w=f_{\theta }(x,y_w)$ and $r_l=f_{\theta }(x,y_l)$
2. Compute contrastive loss:

${{ℒ}}_{RM}=-\log \sigma (r_w-r_l)$

This is equivalent to binary cross-entropy on the reward difference.

Regularization: An optional language modeling loss prevents catastrophic forgetting:

${ℒ}={{ℒ}}_{RM}+{{ℒ}}_{LM}$

Related Pages

Implemented By

• Implementation:LLMBook_zh_LLMBook_zh_github_io_LlamaRewardModel

Uses Heuristic

• Heuristic:LLMBook_zh_LLMBook_zh_github_io_Reward_Model_LM_Regularization