Implementation:OpenRLHF OpenRLHF Get llm for sequence regression

Knowledge Sources	OpenRLHF HuggingFace Transformers
Domains	NLP, Model_Loading, Reward_Modeling
Last Updated	2026-02-07 00:00 GMT

Overview

Concrete tool for constructing transformer models with value heads for reward and critic modeling provided by OpenRLHF.

Description

The get_llm_for_sequence_regression factory function dynamically creates a RewardModel or CriticModel class by attaching a linear projection head to a pretrained transformer base model. It supports LoRA, 4-bit quantization, ZeRO-3, MoE models, and optional value head initialization. The model type ("reward" or "critic") determines whether scalar rewards are extracted at the EOS position or per-token values are returned.

Usage

Import and call when loading a reward model for RM training, reward scoring during batch inference, or a critic model in PPO training.

Code Reference

Source Location

Repository: OpenRLHF
File: openrlhf/models/model.py
Lines: L20-167

Signature

def get_llm_for_sequence_regression(
    model_name_or_path: str,       # HuggingFace model ID or local path
    model_type: str,               # "reward" or "critic"
    *,
    param_dtype="bf16",            # Model precision
    load_in_4bit=False,            # Enable NF4 quantization
    lora_rank=0,                   # LoRA rank (0 = disabled)
    lora_alpha=16,                 # LoRA alpha parameter
    target_modules=None,           # LoRA target modules
    lora_dropout=0,                # LoRA dropout
    normalize_reward=False,        # Normalize reward at inference
    attn_implementation="flash_attention_2",
    ds_config: dict = None,        # DeepSpeed ZeRO-3 config
    init_value_head=False,         # Initialize value head weights
    value_head_prefix="score",     # Attribute name for value head
    device_map=None,
    packing_samples=False,
    **kwargs,
) -> nn.Module:

Import

from openrlhf.models import get_llm_for_sequence_regression

I/O Contract

Inputs

Name	Type	Required	Description
model_name_or_path	str	Yes	Pretrained model path or HuggingFace ID
model_type	str	Yes	"reward" or "critic"
init_value_head	bool	No	Initialize value head (True for RM training)
ds_config	dict	No	DeepSpeed config for ZeRO-3 compatibility

Outputs

Name	Type	Description
model	nn.Module	RewardModel or CriticModel with value head

Usage Examples

Load Reward Model for Training

from openrlhf.models import get_llm_for_sequence_regression

reward_model = get_llm_for_sequence_regression(
    "meta-llama/Llama-2-7b-hf",
    "reward",
    init_value_head=True,
    ds_config=strategy.get_ds_train_config(is_actor=False),
)

Load Critic Model for PPO

critic_model = get_llm_for_sequence_regression(
    "path/to/trained_critic",
    "critic",
    normalize_reward=True,
)

Related Pages

Implements Principle

Principle:OpenRLHF_OpenRLHF_Sequence_Regression_Model_Loading

Requires Environment

Uses Heuristic

Heuristic:OpenRLHF_OpenRLHF_Value_Head_ZeRO3_Init_Tip

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment