Principle:Hiyouga LLaMA Factory Reward Modeling

Knowledge Sources	Hiyouga_LLaMA_Factory Training language models to follow instructions with human feedback
Domains	Natural Language Processing, Language Model Alignment, Preference Learning
Last Updated	2026-02-06 19:00 GMT

Overview

A preference modeling technique that trains a scalar reward function on pairwise human preference data using the Bradley-Terry model, producing a reward signal for downstream reinforcement learning alignment.

Description

Reward Modeling (RM) is the process of training a model to predict human preferences over language model outputs. Given a prompt and two candidate responses, the reward model learns to assign higher scalar scores to the response that humans prefer. This trained reward model then serves as the objective function for reinforcement learning-based alignment (e.g., PPO).

Reward modeling occupies a critical position in the RLHF pipeline:

1. Supervised Fine-Tuning produces an initial instruction-following model.
2. Reward Modeling trains a preference predictor on human comparison data.
3. RL Optimization (PPO) uses the reward model to improve the policy.

The key properties of reward modeling include:

• Bradley-Terry framework: Preferences are modeled as pairwise comparisons under the Bradley-Terry model, where the probability of preferring one response over another is a function of their reward difference.
• Value head architecture: A linear layer is appended to the language model that maps the final hidden state to a scalar reward score.
• Score extraction: The reward is read from the last non-padding token position, capturing the model's overall assessment of the complete response.
• Pairwise training: Each training example consists of a chosen and rejected response for the same prompt.

Usage

Use reward modeling when you want to:

• Train a reward signal for PPO-based RLHF alignment.
• Evaluate and rank model-generated responses according to human preferences.
• Build a preference classifier that can score arbitrary text outputs.
• Create a reranking system for selecting among multiple candidate generations.

Reward modeling requires pairwise preference data where annotators have indicated which of two responses is preferred for a given prompt.

Theoretical Basis

Bradley-Terry Preference Model

The Bradley-Terry model expresses the probability that response $y_w$ is preferred over $y_l$ given prompt $x$ as:

$P(y_w\succ y_l\mid x)=\sigma (r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l))$

where $r_{\varphi }(x,y)$ is the scalar reward assigned by the model with parameters $\varphi$, and $\sigma$ is the sigmoid function.

Training Objective

The reward model is trained by maximizing the log-likelihood of the observed preferences, yielding the pairwise ranking loss:

${{ℒ}}_{\text{RM}}(\varphi )=-{\mathbb{𝔼}}_{(x,y_w,y_l)}[\log \sigma (r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l))]$

This is equivalent to a binary cross-entropy loss where the model must predict that the chosen response has a higher reward than the rejected response. The gradient pushes the chosen score up and the rejected score down.

Score Extraction

Given an input sequence of length $T$ with attention mask $m$, the reward score is extracted at the last valid (non-padding) token position:

$r_{\varphi }(x,y)={\mathbf{𝐰}}^{\mathrm{\top }}{\mathbf{𝐡}}_{\ell }+b$

where $\ell =\sum _{t=1}^T{m}_t-1$ is the index of the last non-padding token, ${\mathbf{𝐡}}_{\ell }$ is the hidden state at that position, and $\mathbf{𝐰},b$ are the value head parameters. This design ensures the reward reflects the model's assessment after processing the entire prompt-response pair.

Evaluation

Reward model quality is measured by pairwise accuracy: the fraction of evaluation examples where the model assigns a higher score to the chosen response than to the rejected response:

$\text{Accuracy}={\mathbb{𝔼}}_{(x,y_w,y_l)}\left[\mathbf{𝟏}[r_{\varphi }(x,y_w)>r_{\varphi }(x,y_l)]\right]$

Related Pages

• Implementation:Hiyouga_LLaMA_Factory_RM_Trainer
• Implementation:Hiyouga_LLaMA_Factory_RM_Workflow