Principle:Huggingface Open r1 Reward Function Design

Metadata

Field	Value
Sources	Paper (DeepSeek-R1 https://arxiv.org/abs/2501.12948), Paper (Demystifying Long Chain-of-Thought Reasoning https://arxiv.org/abs/2502.03373), Paper (DAPO https://arxiv.org/abs/2503.14476)
Domains	Reinforcement_Learning, NLP
Last Updated	2026-02-08 00:00 GMT

Overview

A reward engineering framework that combines multiple programmatic scoring functions to guide reinforcement learning of language models toward correct, well-formatted, and efficient reasoning.

Description

In RL-based LLM training, reward functions define the training signal. Unlike human feedback (RLHF), programmatic rewards enable scalable, reproducible training. This principle covers designing multi-reward systems where different functions score different quality aspects:

• Accuracy - Mathematical correctness verification via symbolic parsing and comparison
• Format compliance - Proper use of <think> and <answer> tags
• Reasoning quality - Step-by-step structure with identifiable reasoning steps
• Efficiency - Length-based rewards derived from Kimi 1.5, rewarding shorter correct solutions more highly
• Repetition penalty - N-gram diversity scoring from the Demystify paper, penalizing degenerate repetitive outputs
• Code correctness - Execution-based scoring where generated code is run against test cases
• Overlong punishment - Soft penalty for outputs exceeding length thresholds, from DAPO

The key design choices are:

1. How to combine multiple reward signals - Each reward function produces an independent score; the GRPO trainer receives a list of reward functions and computes advantages from each. Weights can be assigned via configuration.
2. How to handle unparseable or unverifiable examples - Reward functions return None to signal that a sample should be skipped rather than penalized, preventing noisy gradients from malformed outputs.
3. How to configure reward function parameters via YAML configs - Factory functions accept parameters (e.g., min/max length for cosine scaling, n-gram size for repetition penalty) from a centralized script arguments object, allowing tuning without code changes.

Usage

Use this principle when designing or configuring reward signals for GRPO training. The registry pattern allows selecting reward functions by name in config files without code changes. When adding a new reward function:

1. Define the function following the standard signature: accepting completions: list[list[dict]] and optional kwargs, returning list[float|None].
2. Register it in the REWARD_FUNCS_REGISTRY dictionary.
3. Reference it by name in the training YAML config under reward_funcs.

Theoretical Basis

Multi-Reward Composition

Multiple reward functions are evaluated independently for each completion. The GRPO algorithm computes advantages from each reward signal. The total reward for a completion is the combination of all active reward functions:

# Pseudocode: Multi-reward evaluation
total_rewards = []
for reward_fn in reward_funcs:
    scores = reward_fn(completions=completions, **column_kwargs)
    total_rewards.append(scores)
# GRPO trainer combines these internally for advantage computation

Accuracy Verification via Symbolic Math

Accuracy rewards use a parse-then-verify approach. The answer is extracted from the completion, parsed into a symbolic representation, and compared against the gold answer:

# Pseudocode: Accuracy verification
def accuracy_reward(completions, solution, **kwargs):
    rewards = []
    for completion, gold in zip(completions, solution):
        content = extract_answer(completion)
        parsed_answer = parse_math_expression(content)
        parsed_gold = parse_math_expression(gold)
        if parsed_answer is None or parsed_gold is None:
            rewards.append(None)  # skip unparseable
        elif verify_equivalent(parsed_answer, parsed_gold):
            rewards.append(1.0)
        else:
            rewards.append(0.0)
    return rewards

Cosine-Scaled Length Reward

From Kimi 1.5, shorter correct solutions receive higher reward via cosine scaling. The reward is a function of completion length relative to configured min/max bounds:

# Pseudocode: Cosine-scaled length reward
# Given: min_length, max_length, completion_length
# Normalize length to [0, 1] range
ratio = (completion_length - min_length) / (max_length - min_length)
ratio = clamp(ratio, 0.0, 1.0)
# Cosine scaling: shorter = higher reward
reward = cos(ratio * pi) * 0.5 + 0.5
# Only applied to correct completions; incorrect get 0.0

N-Gram Repetition Penalty

From the Demystify paper, repetitive outputs are penalized based on n-gram diversity:

# Pseudocode: Repetition penalty
# Given: text, ngram_size, max_penalty
tokens = tokenize(text)
ngrams = extract_ngrams(tokens, ngram_size)
total_ngrams = len(ngrams)
unique_ngrams = len(set(ngrams))

if total_ngrams == 0:
    penalty = 0.0
else:
    repetition_ratio = 1.0 - (unique_ngrams / total_ngrams)
    penalty = repetition_ratio * max_penalty

reward = -penalty  # negative reward for repetition

Soft Overlong Punishment (DAPO)

From the DAPO paper, a soft penalty is applied when completions exceed a target length, scaling smoothly rather than applying a hard cutoff:

# Pseudocode: Soft overlong punishment
# Given: completion_length, max_length, penalty_scale
if completion_length > max_length:
    overshoot = (completion_length - max_length) / max_length
    punishment = -penalty_scale * overshoot
else:
    punishment = 0.0
reward = punishment

Related Pages

• Implementation:Huggingface_Open_r1_Get_Reward_Funcs