Principle:OpenRLHF OpenRLHF Process Reward Dataset Construction
| Knowledge Sources | |
|---|---|
| Domains | Data_Processing, Reward_Modeling |
| Last Updated | 2026-02-07 10:40 GMT |
Overview
Data preparation technique that tokenizes reasoning chains and assigns step-level labels at placeholder token positions for PRM training.
Description
Process reward dataset construction prepares training data for process reward models by aligning step-level reward labels with designated placeholder tokens in the input sequence. Input sequences contain special placeholder tokens at reasoning step boundaries. Each placeholder position receives either a hard label (token ID for "+" or "-") or a soft label (float reward value). The construction handles truncation carefully: when the input is truncated to max_length, trailing placeholder tokens and their corresponding labels are also removed to maintain alignment.
Usage
Use this data construction method when building datasets for process reward model (PRM) training. The input data must contain tokenized reasoning chains with placeholder tokens marking step boundaries, and a corresponding list of per-step labels.
Theoretical Basis
The data format encodes step-level supervision into a sequence classification problem:
Pseudo-code Logic:
# Abstract data construction (NOT actual implementation)
input_tokens = tokenize(reasoning_chain_with_placeholders)
labels = full(-100, len(input_tokens)) # Ignore all positions by default
# Assign step labels only at placeholder positions
placeholder_mask = (input_tokens == placeholder_token_id)
labels[placeholder_mask] = step_reward_labels # "+" or "-" token IDs, or float values
# Handle truncation: if input is cut to max_length,
# ensure labels are also truncated to match remaining placeholders
num_remaining = placeholder_mask.sum()
labels[placeholder_mask] = step_reward_labels[:num_remaining]