Principle:Intel Ipex llm DPO Data Preparation
| Knowledge Sources | |
|---|---|
| Domains | NLP, RLHF, Data_Processing |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
Methodology for preparing preference pair datasets in ChatML format for Direct Preference Optimization training.
Description
DPO Data Preparation converts raw preference datasets (containing question, system prompt, chosen, and rejected columns) into the ChatML format required by DPO training. Each example is transformed into a prompt (system + user message formatted via tokenizer.apply_chat_template), a chosen completion, and a rejected completion. The dataset columns are replaced with the formatted versions.
Usage
Use this principle when preparing preference pair datasets for DPO fine-tuning. The input dataset must have columns: system, question, chosen, rejected. The Intel/orca_dpo_pairs dataset is the reference example.
Theoretical Basis
DPO requires preference pairs (y_w, y_l) for each prompt x:
# Abstract data format (NOT real implementation)
# Input: {"system": "...", "question": "...", "chosen": "...", "rejected": "..."}
# Output: {"prompt": "<system> + <user message>", "chosen": "<preferred>", "rejected": "<dispreferred>"}
# ChatML format:
# <|im_start|>system\n{system}<|im_end|>
# <|im_start|>user\n{question}<|im_end|>
# <|im_start|>assistant\n