Principle:Intel Ipex llm DPO Training

Knowledge Sources	Direct Preference Optimization TRL Documentation IPEX-LLM
Domains	NLP, RLHF, Training
Last Updated	2026-02-09 00:00 GMT

Overview

Methodology for training language models with Direct Preference Optimization using the TRL DPOTrainer on Intel XPU.

Description

Direct Preference Optimization (DPO) provides a simpler alternative to RLHF by directly optimizing the policy model using preference pairs. Unlike PPO-based RLHF, DPO does not require a separate reward model. Instead, it uses the preference pairs and the reference model to compute a classification loss that directly aligns the policy with human preferences. The TRL library's DPOTrainer handles the training loop, loss computation, and integration with the DPOConfig training arguments.

Usage

Use this after loading both the policy model (with LoRA) and the reference model, and after formatting the preference dataset. DPO is preferred when you have explicit preference pairs (chosen vs rejected) and want simpler training than PPO-based RLHF.

Theoretical Basis

DPO loss:

$L_{DPO}({\pi }_{\theta };{\pi }_{ref})=-{\mathbb{𝔼}}_{(x,y_w,y_l)\sim D}[\log \sigma (\beta \log \frac{{\pi }_{\theta }(y_w|x)}{{\pi }_{ref}(y_w|x)}-\beta \log \frac{{\pi }_{\theta }(y_l|x)}{{\pi }_{ref}(y_l|x)})]$

Where:

• ${\pi }_{\theta }$ is the policy (trainable model)
• ${\pi }_{ref}$ is the reference (frozen model)
• $\beta$ controls deviation from reference (default 0.1)
• $y_w,y_l$ are chosen and rejected completions

Related Pages

Implemented By

• Implementation:Intel_Ipex_llm_DPOTrainer_Usage

Uses Heuristic

• Heuristic:Intel_Ipex_llm_QLoRA_Training_Hyperparameters