Principle:Huggingface Trl GRPO Argument Configuration
| Property | Value |
|---|---|
| Principle Name | GRPO Argument Configuration |
| Library | Huggingface TRL |
| Category | Configuration / Online RL Training |
| Paper | DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models |
| Related Papers | DAPO, Dr. GRPO, DeepSeek-R1 |
Overview
Description
Group Relative Policy Optimization (GRPO) is an online reinforcement learning algorithm for fine-tuning large language models. Unlike supervised fine-tuning, GRPO requires the model to generate completions during training, score them with reward functions, and then update the policy based on group-relative advantages. This online generation-and-scoring loop introduces a large set of configuration parameters beyond what standard training arguments provide.
The GRPO Argument Configuration principle covers how the training hyperparameters for this online RL pipeline are organized into a single, type-safe dataclass that extends Hugging Face's TrainingArguments. This dataclass, GRPOConfig, groups parameters into logical categories: model and reference model settings, data preprocessing, generation control, vLLM acceleration, training loss formulation, and logging.
A separate companion dataclass, GRPOScriptArguments, extends ScriptArguments to add reward-function selection parameters specific to the GRPO training script.
Usage
The configuration objects are parsed from YAML config files and/or command-line arguments using the TrlParser, a subclass of Hugging Face's HfArgumentParser. The parser supports:
- YAML-based configuration with environment variable injection via an
env:section - Command-line overrides that take priority over YAML defaults
- Multiple dataclass types parsed simultaneously (script args, training config, model config, dataset config)
Theoretical Basis
GRPO was introduced in the DeepSeekMath paper (Shao et al., 2024). The algorithm generates G completions per prompt, computes rewards, derives group-relative advantages (subtracting the group mean and dividing by the group standard deviation), and updates the policy with a clipped surrogate objective similar to PPO but without a value function.
Key configurable theoretical quantities include:
- num_generations (G): The number of completions sampled per prompt. The paper uses groups of 64. A minimum of 2 is required for advantage normalization.
- beta: The KL divergence coefficient penalizing drift from the reference model. Setting beta to 0.0 disables KL regularization entirely (and skips loading the reference model), as recommended for reasoning-focused training. DeepSeek-R1 uses a value of 0.001.
- epsilon and epsilon_high: Clipping bounds for the importance-sampling ratio in the surrogate objective. The DAPO paper recommends asymmetric clipping with epsilon=0.2 and epsilon_high=0.28.
- loss_type: Selects the loss normalization strategy. Options include
grpo(per-sequence),dr_grpo(global constant),dapo(global active tokens, the default),bnpo(local active tokens),cispo(MiniMax-M1 clipped IS weights), andsapo(soft adaptive). - scale_rewards: Controls whether advantage normalization divides by within-group standard deviation (
group), batch-level standard deviation (batch), or performs no scaling (none). Dr. GRPO recommends no scaling. - temperature: Sampling temperature for generation. Higher values increase diversity in the generated completions, which is critical for exploration in RL.
- use_vllm: Whether to offload generation to a vLLM server for significant speed improvements. When enabled, additional vLLM-specific parameters control server mode, colocated mode, tensor parallelism, memory utilization, and importance-sampling correction.
The TrlParser validates constraints at parse time, including ensuring the effective batch size is divisible by num_generations, the generation batch size is properly aligned, and mutually exclusive parameters (e.g., generation_batch_size vs steps_per_generation) are not both specified.