Principle:OpenRLHF OpenRLHF Unpaired Preference Dataset Construction

Knowledge Sources	KTO: Model Alignment as Prospect Theoretic Optimization
Domains	Data_Processing, Alignment
Last Updated	2026-02-07 10:40 GMT

Overview

Data preparation technique that formats independently labeled preference samples with unmatched pairs for KL divergence estimation.

Description

Unpaired preference dataset construction prepares training data where each sample has an independent binary label (desirable or undesirable) rather than being part of a chosen/rejected pair. The critical transformation is the batch doubling step: for each batch of N matched (prompt, response, label) triples, N additional unmatched pairs are created by pairing prompt[i] with response[(i+1) % N]. These unmatched pairs (labeled with -1) provide the reference distribution needed to estimate the KL divergence term in the KTO loss function.

Usage

Use this data construction method when preparing datasets for KTO training. The input data must have independently labeled samples (e.g., from thumbs-up/down feedback) rather than paired preferences. Each sample requires a prompt, response, and binary desirability label.

Theoretical Basis

The key insight is that KTO requires estimating $KL({\pi }_{\theta }||{\pi }_{ref})$ to compute the reference point $z_{ref}$. Since directly computing this KL requires on-policy samples, the unmatched pairs provide an efficient approximation:

Pseudo-code Logic:

# Abstract batch construction (NOT actual implementation)
matched_pairs = [(prompt_i, response_i, label_i) for i in range(N)]

# Create unmatched pairs for KL estimation
unmatched_pairs = [(prompt_i, response_{(i+1)%N}, -1) for i in range(N)]

# Final batch has 2N samples
batch = matched_pairs + unmatched_pairs
# Front half: supervised preference learning
# Back half: KL divergence estimation

Related Pages

• Implementation:OpenRLHF_OpenRLHF_UnpairedPreferenceDataset_init