Principle:Eric mitchell Direct preference optimization Preference Data Format
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, NLP, Preference_Learning |
| Last Updated | 2026-02-08 02:00 GMT |
Overview
A canonical data format contract for preference datasets that standardizes how prompts, responses, preference orderings, and SFT targets are represented.
Description
The preference data format defines a standard interface that all dataset loaders must produce. This canonical format enables the batch data pipeline to work uniformly across different preference datasets (Anthropic HH, Stanford Human Preferences, StackExchange, or custom datasets).
The format is a dictionary keyed by prompt strings, where each value contains:
- responses: A list of all response texts for that prompt
- pairs: A list of (winner_index, loser_index) tuples indicating preference orderings
- sft_target: The single best response for supervised fine-tuning
This separation allows the same dataset to be used for both SFT training (using only sft_target) and DPO training (using all preference pairs).
Usage
Understand this format when implementing custom dataset loaders or when debugging data pipeline issues. All three built-in loaders (get_hh, get_shp, get_se) produce data in this format, and the dispatcher validates the keys with an assertion.
Theoretical Basis
Preference-based training requires structured preference data. The canonical format supports the general case of K-wise preferences (multiple responses with pairwise orderings) while also accommodating the simpler case of binary preferences (single chosen/rejected pair).
Pseudo-code:
# Abstract canonical format (NOT actual implementation)
data = {
"\n\nHuman: What is 2+2?\n\nAssistant:": {
"responses": [" 4.", " 22.", " Four."],
"pairs": [(0, 1), (2, 1)], # response 0 > 1, response 2 > 1
"sft_target": " 4.", # best response for SFT
},
# ... more prompts
}