Principle:OpenRLHF OpenRLHF Prompt Dataset Construction

Knowledge Sources	Training language models to follow instructions
Domains	Data_Processing, Reinforcement_Learning
Last Updated	2026-02-07 00:00 GMT

Overview

A dataset preparation technique that formats prompts for on-policy generation in reinforcement learning and iterative training workflows.

Description

Prompt Dataset Construction prepares prompts (without responses) for RL training. Unlike SFT datasets that contain both prompts and responses, prompt datasets provide only the input that will be given to the policy model for generation. The generated responses are then scored by a reward model or rule-based function to create training signal.

It also supports label tracking for reinforced fine-tuning (e.g., correct answers for math problems) and data source tracking for per-source reward analysis.

Usage

Use when setting up PPO, GRPO, rejection sampling, or iterative DPO training. The prompts are used for on-policy generation.

Theoretical Basis

On-policy RL requires fresh generations from the current policy at each training step. The prompt dataset provides the starting points for generation:

1. Sample prompt $x\sim D_{prompts}$
2. Generate response $y\sim {\pi }_{\theta }(\cdot |x)$
3. Score response $r=R(x,y)$
4. Use $(x,y,r)$ for policy gradient update

Related Pages

Implemented By

• Implementation:OpenRLHF_OpenRLHF_PromptDataset_init