Principle:OpenRLHF OpenRLHF Rejection Sampling

Knowledge Sources	Llama 2: Open Foundation and Fine-Tuned Chat Models
Domains	Alignment, Data_Processing
Last Updated	2026-02-07 00:00 GMT

Overview

An iterative alignment technique that generates multiple responses per prompt, selects the highest-reward response, and retrains the model on this filtered data.

Description

Rejection Sampling (also called Best-of-N sampling) generates multiple candidate responses from the policy model for each prompt, scores them with a reward model, and selects the best response. The selected (prompt, best_response) pairs form a curated SFT dataset used to retrain the model. This process can be repeated iteratively, progressively improving the model.

Usage

Use as an alternative to PPO when simplicity is preferred. The workflow cycles: generate N responses per prompt, score all, keep the best, retrain with SFT. Requires a trained reward model.

Theoretical Basis

For each prompt $x$:

1. Generate $N$ responses: $y_1,...,y_N\sim {\pi }_{\theta }(\cdot |x)$
2. Score each: $r_i=R(x,y_i)$
3. Select best: $y^{*}=\arg \underset{i}{\max }{r}_i$
4. Retrain: minimize $-\log {\pi }_{\theta }(y^{*}|x)$

This implicitly optimizes: ${\mathbb{𝔼}}_x[\underset{y\in \{y_1,...,y_N\}}{\max }R(x,y)]$

Related Pages

Implemented By

• Implementation:OpenRLHF_OpenRLHF_Rejection_sampling_processor