Principle:OpenRLHF OpenRLHF KTO Training

Knowledge Sources	KTO: Model Alignment as Prospect Theoretic Optimization
Domains	Alignment, Reinforcement_Learning, Optimization
Last Updated	2026-02-07 10:40 GMT

Overview

Alignment technique that trains language models using unpaired preference data based on Kahneman-Tversky prospect theory.

Description

Kahneman-Tversky Optimization (KTO) is a human alignment method that operates on unpaired preference data, where each sample is independently labeled as desirable or undesirable. Unlike DPO which requires paired (chosen, rejected) examples, KTO leverages the insight from prospect theory that humans value losses more than equivalent gains. The loss function applies asymmetric weighting: undesirable outputs are penalized more heavily than desirable outputs are rewarded. A KL divergence term estimated from unmatched prompt-response pairs regularizes the policy to stay close to the reference model.

Usage

Use KTO training when you have preference data that is not naturally paired. This is common when feedback is collected independently (e.g., thumbs up/down on individual responses) rather than as side-by-side comparisons. KTO is a simpler alternative to DPO when paired data is unavailable, while still achieving competitive alignment quality.

Theoretical Basis

The KTO loss decomposes into desirable and undesirable terms:

$L_{K T O} = w_{d} \cdot 𝔼_{(x, y) \sim D_{d e s i r a b l e}} [1 - σ (β \cdot (r_{θ} (x, y) - z_{r e f}))] + w_{u} \cdot 𝔼_{(x, y) \sim D_{u n d e s i r a b l e}} [1 - σ (β \cdot (z_{r e f} - r_{θ} (x, y)))]$

Where:

$r_{θ} (x, y) = \log π_{θ} (y | x) - \log π_{r e f} (y | x)$ is the implicit reward
$z_{r e f}$ is the KL divergence estimated from unmatched pairs
$β$ controls regularization strength
$w_{d}, w_{u}$ are loss weights for desirable and undesirable samples

Pseudo-code Logic:

# Abstract algorithm (NOT actual implementation)
policy_logps = compute_logprobs(policy_model, inputs)
ref_logps = compute_logprobs(ref_model, inputs)
kl_estimate = compute_kl_from_unmatched_pairs(policy_logps, ref_logps)

for sample in batch:
    reward = policy_logps[sample] - ref_logps[sample]
    if sample.label == desirable:
        loss += w_d * (1 - sigmoid(beta * (reward - kl_estimate)))
    else:
        loss += w_u * (1 - sigmoid(beta * (kl_estimate - reward)))

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment