Implementation:Hiyouga LLaMA Factory DPO En Demo Data

Knowledge Sources	Hiyouga_LLaMA_Factory
Domains	NLP, Training_Data
Last Updated	2026-02-06 19:00 GMT

Overview

dpo_en_demo.json provides English preference data with chosen and rejected response pairs in ShareGPT format for demonstrating and testing Direct Preference Optimization (DPO) and reward modeling workflows in LLaMA Factory.

Description

The file contains a JSON array of conversation records, each consisting of a conversations list (multi-turn dialogue history in ShareGPT format with "from" and "value" fields), a chosen object (the preferred assistant response), and a rejected object (the dispreferred assistant response). Conversations may include system messages, and the chosen/rejected responses represent the human preference signal used for DPO training. Each response object contains "from": "gpt" and the response text in "value".

This dataset is registered in dataset_info.json with "ranking": true and ShareGPT formatting, mapping the conversations, chosen, and rejected columns accordingly.

Usage

This demo dataset is used for quick testing of DPO training pipelines and reward model training. Users reference it by name (dpo_en_demo) with --stage dpo or --stage rm in training configuration to verify that preference data loading and pairwise training work correctly.

Code Reference

Source Location

• Repository: Hiyouga_LLaMA_Factory
• File: data/dpo_en_demo.json

Data Format

[
  {
    "conversations": [
      {
        "from": "human",
        "value": "Hi! I'd like to create a new language game simulating the first person perspective of a character named Angela."
      }
    ],
    "chosen": {
      "from": "gpt",
      "value": "That sounds like a fun and engaging idea! Here are some tips to help you create the game..."
    },
    "rejected": {
      "from": "gpt",
      "value": "Hello! I'd be happy to help you create a language game simulating the first-person perspective..."
    }
  }
]

I/O Contract

Schema

Field	Type	Required	Description
conversations	array	Yes	List of conversation turns, each with "from" (human/system/gpt) and "value" (message text)
chosen	object	Yes	The preferred response with "from": "gpt" and "value" containing the chosen text
rejected	object	Yes	The dispreferred response with "from": "gpt" and "value" containing the rejected text

Conversation Turn Schema

Field	Type	Required	Description
from	string	Yes	Role identifier: "human", "gpt", or "system"
value	string	Yes	The message content

Dataset Registry Entry

Property	Value
Key	dpo_en_demo
file_name	dpo_en_demo.json
formatting	sharegpt
ranking	true
columns.messages	conversations
columns.chosen	chosen
columns.rejected	rejected
Lines	7226

Usage Examples

# Reference the dataset in a LLaMA Factory training config for DPO
# llamafactory-cli train \
#     --dataset dpo_en_demo \
#     --stage dpo \
#     --model_name_or_path meta-llama/Llama-2-7b-hf \
#     --output_dir output/dpo_demo

# Or for reward model training
# llamafactory-cli train \
#     --dataset dpo_en_demo \
#     --stage rm \
#     --model_name_or_path meta-llama/Llama-2-7b-hf \
#     --output_dir output/rm_demo

# Loading the data manually for inspection
import json

with open("data/dpo_en_demo.json", "r", encoding="utf-8") as f:
    data = json.load(f)

print(f"Number of preference pairs: {len(data)}")
sample = data[0]
print(f"Conversation turns: {len(sample['conversations'])}")
print(f"Chosen response length: {len(sample['chosen']['value'])}")
print(f"Rejected response length: {len(sample['rejected']['value'])}")

Related Pages

• Hiyouga_LLaMA_Factory_DPO_Zh_Demo_Data - Chinese version of the DPO demo dataset
• Hiyouga_LLaMA_Factory_KTO_En_Demo_Data - Alternative preference format using binary labels
• Hiyouga_LLaMA_Factory_Dataset_Info_Registry - Central dataset registry that indexes this file
• Hiyouga_LLaMA_Factory_Alpaca_En_Demo_Data - English SFT demo data (non-preference)