Principle:Hiyouga LLaMA Factory Direct Preference Optimization

Knowledge Sources	Hiyouga_LLaMA_Factory Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Domains	Natural Language Processing, Language Model Alignment, Preference Learning
Last Updated	2026-02-06 19:00 GMT

Overview

A preference-based alignment technique that optimizes a language model directly on human preference data without requiring a separate reward model or reinforcement learning loop.

Description

Direct Preference Optimization (DPO) is an alignment method introduced by Rafailov et al. (2023) that reframes the RLHF objective as a simple classification problem over preference pairs. Rather than first training a reward model and then using PPO to optimize the policy against that reward, DPO derives a closed-form mapping between the optimal policy and the reward function. This allows the preference loss to be expressed directly in terms of the policy model's log-probabilities.

DPO is significant in the ML landscape because it:

• Eliminates the reward model: No separate reward model needs to be trained or maintained during alignment.
• Removes RL complexity: No PPO clipping, value function estimation, or advantage computation is needed.
• Maintains stability: The implicit KL-divergence constraint against a reference model prevents the policy from deviating too far from the pretrained distribution.
• Supports multiple loss variants: The framework naturally extends to IPO (identity preference optimization), ORPO (odds ratio preference optimization), SimPO (simple preference optimization), and BCO (binary classifier optimization).

The method requires pairwise preference data where, for each prompt, a chosen (preferred) response and a rejected (dispreferred) response are provided.

Usage

Use DPO when you want to:

• Align a language model to human preferences after supervised fine-tuning.
• Avoid the complexity and instability of PPO-based RLHF.
• Work with pairwise preference datasets (chosen vs. rejected responses).
• Experiment with preference optimization variants (IPO, ORPO, SimPO) using a unified framework.

DPO is most effective when high-quality pairwise preference data is available and the SFT model already produces reasonable outputs.

Theoretical Basis

Core DPO Objective

DPO starts from the observation that the optimal policy under a KL-constrained reward maximization objective satisfies:

$r(x,y)=\beta \log \frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}+\beta \log Z(x)$

where $r(x,y)$ is the implicit reward, ${\pi }_{\theta }$ is the policy, ${\pi }_{\text{ref}}$ is the reference model, $\beta$ is the KL penalty coefficient, and $Z(x)$ is the partition function. Substituting this into the Bradley-Terry preference model yields the DPO loss:

${{ℒ}}_{\text{DPO}}(\theta )=-{\mathbb{𝔼}}_{(x,y_w,y_l)}[\log \sigma (\beta \log \frac{{\pi }_{\theta }(y_w\mid x)}{{\pi }_{\text{ref}}(y_w\mid x)}-\beta \log \frac{{\pi }_{\theta }(y_l\mid x)}{{\pi }_{\text{ref}}(y_l\mid x)})]$

where $y_w$ is the chosen (winning) response, $y_l$ is the rejected (losing) response, and $\sigma$ is the sigmoid function.

Loss Variants

The framework supports several loss types:

IPO (Identity Preference Optimization) uses average log-probabilities rather than summed log-probabilities:

${{ℒ}}_{\text{IPO}}={(\log \frac{{\pi }_{\theta }(y_w\mid x)}{{\pi }_{\text{ref}}(y_w\mid x)}-\log \frac{{\pi }_{\theta }(y_l\mid x)}{{\pi }_{\text{ref}}(y_l\mid x)}-\frac{1}{2\beta })}^2$

ORPO (Odds Ratio Preference Optimization) is reference-free and combines SFT with an odds ratio penalty:

${{ℒ}}_{\text{ORPO}}=-\log {\pi }_{\theta }(y_w\mid x)+\beta \cdot (-\log \sigma (\log \frac{\text{odds}(y_w)}{\text{odds}(y_l)}))$

SimPO (Simple Preference Optimization) is reference-free and uses a length-normalized margin:

${{ℒ}}_{\text{SimPO}}=-\log \sigma \left(\beta (\overline{r}(y_w)-\overline{r}(y_l)-\gamma )\right)$

where $\gamma$ is a target reward margin and $\overline{r}$ denotes length-normalized log-probabilities.

Auxiliary SFT Loss

An optional auxiliary SFT loss on the chosen responses can be added to prevent catastrophic forgetting:

${{ℒ}}_{\text{total}}={{ℒ}}_{\text{DPO}}+{\gamma }_{\text{ftx}}\cdot {{ℒ}}_{\text{SFT}}(y_w)$

where ${\gamma }_{\text{ftx}}$ controls the weight of the SFT regularization term.

Reference Model

When a reference model is used (use_ref_model=True), DPO computes log-probabilities from both the policy and a frozen copy of the original model. When using LoRA, the reference model can be implicitly obtained by disabling the adapter layers, avoiding the need to load a separate model into memory.

Related Pages

• Implementation:Hiyouga_LLaMA_Factory_DPO_Trainer
• Implementation:Hiyouga_LLaMA_Factory_DPO_Workflow
• Implementation:Hiyouga_LLaMA_Factory_Pairwise_Processor