Principle:Allenai Open instruct DPO Loss

Knowledge Sources	DPO SimPO WPO
Domains	Machine Learning, Optimization, Preference Learning
Last Updated	2026-02-07 00:00 GMT

Overview

The DPO loss is a preference optimization objective that directly optimizes a language model policy to align with human preferences, bypassing the need to train an explicit reward model.

Description

Direct Preference Optimization (DPO) reformulates the reinforcement learning from human feedback (RLHF) objective into a simple classification-style loss over preference pairs. Given a prompt $x$, a preferred (chosen) response $y_w$, and a dispreferred (rejected) response $y_l$, DPO derives a closed-form loss that implicitly optimizes the same objective as RLHF with a KL-divergence constraint.

The key insight is that the optimal policy under the constrained RLHF objective can be expressed in terms of the reward function and the reference policy. By rearranging this relationship, the reward function can be eliminated entirely, yielding a loss that depends only on the policy and reference model log-probabilities.

Standard DPO Loss:

The standard DPO loss computes the sum of log-probabilities over response tokens, forming log-ratios between the policy and reference models for both chosen and rejected responses, then applies a sigmoid function.

DPO-Norm:

DPO-Norm is a variant that uses the average log-probability (per token) instead of the sum. This normalizes for response length, preventing the model from preferring shorter responses simply because they have higher summed log-probabilities.

SimPO (Simple Preference Optimization):

SimPO removes the reference model entirely, using only the policy's average log-probabilities. It introduces a margin term $\gamma$ to create a target gap between chosen and rejected response scores. This eliminates the need to cache or compute reference model logprobs.

WPO (Weighted Preference Optimization):

WPO extends DPO by weighting the loss based on the policy model's confidence. The weight is computed from the average log-probabilities of both chosen and rejected responses, clamped to [0, 1]. This focuses the training signal on examples where the model is less confident.

Label Smoothing:

All variants support label smoothing, which softens the binary preference signal by mixing in a small probability of the "wrong" preference direction:

${ℒ}=-(1-ϵ)\log \sigma (\beta \cdot \text{logits})-ϵ\log \sigma (-\beta \cdot \text{logits})$

where $ϵ$ is the label smoothing parameter.

Usage

Use the DPO loss when:

• You have paired preference data (chosen vs. rejected responses for the same prompt).
• You want to align a language model without training a separate reward model.
• You need fine-grained control over the loss variant (standard, normalized, reference-free, or weighted).

Theoretical Basis

DPO Derivation:

Starting from the constrained RLHF objective:

$\underset{\pi }{\max }{\mathbb{𝔼}}_{x\sim {𝒟},y\sim \pi (\cdot |x)}[r(x,y)]-\beta \text{KL}[\pi (\cdot |x)\Vert {\pi }_{\text{ref}}(\cdot |x)]$

the optimal policy is:

${\pi }^{*}(y|x)=\frac{1}{Z(x)}{\pi }_{\text{ref}}(y|x)\exp \left(\frac{r(x,y)}{\beta }\right)$

Solving for the reward:

$r(x,y)=\beta \log \frac{{\pi }^{*}(y|x)}{{\pi }_{\text{ref}}(y|x)}+\beta \log Z(x)$

Substituting into the Bradley-Terry preference model and noting that $Z(x)$ cancels:

${{ℒ}}_{\text{DPO}}({\pi }_{\theta };{\pi }_{\text{ref}})=-\mathbb{𝔼}[\log \sigma \left(\beta (\log \frac{{\pi }_{\theta }(y_w|x)}{{\pi }_{\text{ref}}(y_w|x)}-\log \frac{{\pi }_{\theta }(y_l|x)}{{\pi }_{\text{ref}}(y_l|x)})\right)]$

Implicit Rewards:

The DPO framework also defines implicit reward metrics for monitoring training:

Failed to parse (syntax error): {\displaystyle \text{chosen\_reward} = \beta \left(\log \pi_\theta(y_w|x) - \log \pi_{\text{ref}}(y_w|x)\right)}

Failed to parse (syntax error): {\displaystyle \text{rejected\_reward} = \beta \left(\log \pi_\theta(y_l|x) - \log \pi_{\text{ref}}(y_l|x)\right)}

The reward margin (chosen minus rejected) should increase during training, indicating the policy increasingly prefers the chosen response.

SimPO Loss:

${{ℒ}}_{\text{SimPO}}=-\log \sigma \left(\beta (\overline{\log {\pi }_{\theta }(y_w|x)}-\overline{\log {\pi }_{\theta }(y_l|x)}-\gamma /\beta )\right)$

where $\overline{\log {\pi }_{\theta }(y|x)}$ denotes the length-normalized average log-probability.

Related Pages

Implemented By

• Implementation:Allenai_Open_instruct_DPO_Loss_Function