Implementation:OpenRLHF OpenRLHF Iterative dpo processor

Knowledge Sources	OpenRLHF
Domains	Alignment, Data_Processing
Last Updated	2026-02-07 00:00 GMT

Overview

Concrete tool for constructing on-policy preference pairs from scored generations provided by OpenRLHF.

Description

The iterative_dpo_processor function groups scored generations by prompt and selects the highest-reward response as "chosen" and the lowest-reward response as "rejected" for each prompt. The output format matches the DPO dataset schema with prompt, chosen, and rejected fields.

Usage

Called after batch vLLM generation and batch reward scoring. The output is used to create a RewardDataset(is_dpo=True) for DPO retraining.

Code Reference

Source Location

• Repository: OpenRLHF
• File: openrlhf/utils/processor.py
• Lines: L58-89

Signature

def iterative_dpo_processor(args, objs):
    """
    Create preference pairs from scored generations.

    Args:
        args: CLI arguments (unused)
        objs: List of dicts with keys: "input", "output", "reward"

    Returns:
        List of dicts: [{
            "prompt": str,
            "chosen": str,
            "chosen_reward": float,
            "rejected": str,
            "rejected_reward": float
        }]
    """

Import

from openrlhf.utils.processor import iterative_dpo_processor
# or
from openrlhf.utils.processor import get_processor
processor = get_processor("iter_dpo")

I/O Contract

Inputs

Name	Type	Required	Description
objs	List[Dict]	Yes	Scored generations: [{input, output, reward}, ...]

Outputs

Name	Type	Description
pairs	List[Dict]	Preference pairs: [{prompt, chosen, chosen_reward, rejected, rejected_reward}, ...]

Usage Examples

from openrlhf.utils.processor import get_processor

processor = get_processor("iter_dpo")
preference_pairs = processor(args, scored_generations)
# Each entry has: prompt, chosen, chosen_reward, rejected, rejected_reward

Related Pages

Implements Principle

• Principle:OpenRLHF_OpenRLHF_Iterative_DPO