Environment:Huggingface Open r1 Slurm Cluster

Knowledge Sources	Open R1 Slurm LightEval
Domains	Infrastructure, Evaluation
Last Updated	2026-02-08 00:00 GMT

Overview

A Slurm-managed compute cluster for submitting LightEval benchmark evaluation jobs with automatic GPU allocation and vLLM inference.

Description

This environment defines the Slurm cluster infrastructure required for benchmark evaluation in Open-R1. The evaluation system submits sbatch jobs via a special environment prefix (VLLM_SLURM_PREFIX) that sanitizes the shell environment to avoid conflicts with the parent training job. Each evaluation job runs LightEval with vLLM-accelerated inference on configurable benchmark suites. The Slurm scheduler handles GPU allocation based on model size: models with 30B+ parameters use tensor parallelism across all available GPUs, while smaller models default to 2 GPUs.

Usage

Use this environment for Benchmark Evaluation after training or at each checkpoint via the PushToHubRevisionCallback. It is also used for Pass Rate Filtering parallelization (launching ~88 Slurm shards). The Slurm scheduler must be available and properly configured on the compute cluster.

System Requirements

Category	Requirement	Notes
OS	Linux	Slurm only supports Linux
Scheduler	Slurm	sinfo and sbatch must be available on PATH
Hardware	NVIDIA GPUs	Requested via --gres=gpu:N where N is computed per-model
Network	Shared filesystem	For checkpoint access and result storage across nodes

Dependencies

System Packages

• slurm (scheduler daemons and client tools)
• cuda-toolkit = 12.4

Python Packages

• lighteval (pinned to specific git commit for bug fix)
• vllm == 0.8.5.post1
• torch == 2.6.0
• transformers == 4.52.3

Slurm Scripts

• slurm/evaluate.slurm -- The sbatch script for running LightEval
• scripts/pass_rate_filtering/launch_filtering.sh -- Parallel Slurm launch for pass rate filtering

Credentials

The following environment variables must be accessible within Slurm jobs:

• HF_TOKEN: HuggingFace API token for model download and result upload.
• HOME: Explicitly set in the Slurm prefix to the user's home directory (required because env -i clears all variables).

Quick Install

# Verify Slurm is available
sinfo

# Install LightEval (via Open-R1 dev extras)
GIT_LFS_SKIP_SMUDGE=1 uv pip install -e ".[eval]"

Code Evidence

Slurm availability check from src/open_r1/utils/callbacks.py:28-34:

def is_slurm_available() -> bool:
    # returns true if a slurm queueing system is available
    try:
        subprocess.run(["sinfo"], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        return True
    except FileNotFoundError:
        return False

Special vLLM-Slurm environment prefix from src/open_r1/utils/evaluation.py:14-24:

# We need a special environment setup to launch vLLM from within Slurm training jobs.
# - Reference code: https://github.com/huggingface/brrr/blob/c55ba3505686d690de24c7ace6487a5c1426c0fd/brrr/lighteval/one_job_runner.py#L105
# - Slack thread: https://huggingface.slack.com/archives/C043JTYE1MJ/p1726566494958269
user_home_directory = os.path.expanduser("~")
VLLM_SLURM_PREFIX = [
    "env",
    "-i",
    "bash",
    "-c",
    f"for f in /etc/profile.d/*.sh; do source $f; done; export HOME={user_home_directory}; sbatch ",
]

GPU allocation and tensor parallelism logic from src/open_r1/utils/evaluation.py:77-83:

num_gpus = get_gpu_count_for_vllm(model_name, model_revision)
if get_param_count_from_repo_id(model_name) >= 30_000_000_000:
    tensor_parallel = True
else:
    num_gpus = 2  # Hack while cluster is full
    tensor_parallel = False

Common Errors

Error Message	Cause	Solution
FileNotFoundError: sinfo	Slurm not installed or not on PATH	Install Slurm client tools or add to PATH
sbatch: error: Batch job submission failed	Invalid Slurm configuration or resource unavailable	Check cluster status with sinfo and adjust GPU request
Unknown benchmark {name}	Benchmark not registered in LIGHTEVAL_TASKS	Use one of: math_500, aime24, aime25, gpqa, lcb, lcb_v4, or "all"
vLLM fails inside Slurm job	Environment variables not propagated	The VLLM_SLURM_PREFIX handles this by using env -i and re-sourcing profiles

Compatibility Notes

• Environment isolation: The VLLM_SLURM_PREFIX uses env -i to clear all environment variables before launching sbatch. This is necessary to avoid conflicts between the parent training job's environment (e.g., CUDA_VISIBLE_DEVICES, distributed training vars) and the evaluation job.
• GPU count hack: For models under 30B parameters, the system currently hardcodes 2 GPUs instead of computing the optimal count. This is a temporary workaround documented as "Hack while cluster is full".
• Slurm config tuning: The Slurm configuration in slurm/train.slurm is optimized for the Hugging Face Compute Cluster and may require tweaking for other clusters.
• Benchmarks: When benchmarks=["all"], all registered LightEval tasks are evaluated: math_500, aime24, aime25, gpqa, lcb, lcb_v4.

Related Pages

• Implementation:Huggingface_Open_r1_Run_Benchmark_Jobs