Principle:OpenRLHF OpenRLHF DPO Loss Computation

Knowledge Sources	Direct Preference Optimization IPO: A General Theoretical Paradigm cDPO: Conservative DPO
Domains	Alignment, Loss_Functions
Last Updated	2026-02-07 00:00 GMT

Overview

A family of loss functions that optimize language model policies directly from preference data by comparing log-probability ratios between policy and reference models.

Description

DPO Loss Computation implements the core mathematical operation of Direct Preference Optimization. It computes the implicit reward margin between chosen and rejected responses by comparing the log-probability ratios of the policy model against a frozen reference model. The loss encourages the policy to increase the probability of preferred responses relative to the reference.

Three variants are supported: standard DPO, conservative DPO (with label smoothing), and IPO (Identity Preference Optimization with squared loss).

Usage

Used internally by DPOTrainer. The variant is selected via args.ipo (boolean) and args.label_smoothing (float) parameters.

Theoretical Basis

Standard DPO loss: $L=-\log \sigma \left(\beta (\log \frac{{\pi }_{\theta }(y_w|x)}{{\pi }_{ref}(y_w|x)}-\log \frac{{\pi }_{\theta }(y_l|x)}{{\pi }_{ref}(y_l|x)})\right)$

Conservative DPO (cDPO): Adds label smoothing $ϵ$: $L=-(1-ϵ)\log \sigma (\beta \cdot \mathrm{\Delta })-ϵ\log \sigma (-\beta \cdot \mathrm{\Delta })$

IPO: Replaces sigmoid with squared loss: $L=(\mathrm{\Delta }-\frac{1}{2\beta }{)}^2$

where $\mathrm{\Delta }=\log \frac{{\pi }_{\theta }(y_w|x)}{{\pi }_{ref}(y_w|x)}-\log \frac{{\pi }_{\theta }(y_l|x)}{{\pi }_{ref}(y_l|x)}$

Related Pages

Implemented By

• Implementation:OpenRLHF_OpenRLHF_DPOLoss