Principle:OpenRLHF OpenRLHF SFT Dataset Construction

Knowledge Sources	Training language models to follow instructions with human feedback
Domains	Data_Processing, NLP
Last Updated	2026-02-07 00:00 GMT

Overview

A dataset preparation technique that tokenizes instruction-response pairs with selective loss masking to train only on response tokens.

Description

SFT Dataset Construction converts raw instruction-response data into tokenized training samples where the loss is computed only on the response portion. This selective loss masking prevents the model from being penalized for "generating" the instruction, focusing learning on generating appropriate responses.

The construction supports: (1) single-turn prompt-response pairs, (2) multi-turn conversations with multiple response segments, (3) continued pretraining mode (loss on all tokens), and (4) chat template application for model-specific formatting.

Usage

Use this principle when preparing data for supervised fine-tuning or knowledge distillation. Both SFT and KD workflows use the same dataset format.

Theoretical Basis

Loss masking ensures the cross-entropy loss is only computed on response tokens: $L=-\frac{1}{|R|}\sum _{t\in R}\log P(x_t|x_{<t})$ where $R$ is the set of response token positions.

For multi-turn conversations, the mask includes all assistant response segments while masking all user turns.

Related Pages

Implemented By

• Implementation:OpenRLHF_OpenRLHF_SFTDataset_init