Implementation:Huggingface Transformers BenchmarkRunner Load Model

Knowledge Sources	Transformers
Domains	Benchmarking, Performance, Model Loading
Last Updated	2026-02-13 00:00 GMT

Overview

Concrete tool for loading a model and tokenizer with benchmark-specific settings provided by the HuggingFace Transformers benchmark framework.

Description

BenchmarkRunner.setup_benchmark is a method that prepares a model and tokenizer for a benchmark run. It performs tokenizer initialization (once per model ID), input preparation (tokenizing a fixed prompt to the configured batch size and sequence length), generation configuration setup (including optional CompileConfig and static cache), and model loading via AutoModelForCausalLM.from_pretrained with the specified dtype, attention implementation, kernelization flag, and device placement. The loaded model is set to evaluation mode. The tokenizer's EOS token is reassigned to the padding token to enable open-ended generation without premature stopping.

Usage

Use setup_benchmark before calling run_benchmark for each benchmark configuration. It is called automatically by run_benchmarks for each configuration in a sweep. If running benchmarks manually, call it explicitly before run_benchmark.

Code Reference

Source Location

• Repository: transformers
• File: benchmark_v2/framework/benchmark_runner.py (lines 172-217)

Signature

class BenchmarkRunner:
    def __init__(
        self,
        logger: logging.Logger,
        output_dir: str | None = None,
        branch_name: str | None = None,
        commit_id: str | None = None,
        commit_message: str | None = None,
    ) -> None:
        ...

    def setup_benchmark(self, model_id: str, config: BenchmarkConfig) -> None:
        ...

Import

from benchmark_v2.framework.benchmark_runner import BenchmarkRunner

I/O Contract

Inputs (BenchmarkRunner.__init__)

Name	Type	Required	Description
logger	logging.Logger	Yes	Logger instance for benchmark progress and diagnostic messages.
output_dir	None	No (default: None)	Directory for saving results. Defaults to benchmark_v2/benchmark_results.
branch_name	None	No (default: None)	Git branch name for metadata tagging.
commit_id	None	No (default: None)	Git commit hash. Auto-detected from .git/HEAD if not provided.
commit_message	None	No (default: None)	Git commit message for metadata tagging.

Inputs (setup_benchmark)

Name	Type	Required	Description
model_id	str	Yes	Hugging Face model identifier (e.g., "meta-llama/Llama-3-8B").
config	BenchmarkConfig	Yes	Benchmark configuration specifying dtype, attention implementation, compile settings, etc.

Outputs

Name	Type	Description
(side effects)	N/A	Sets self.model, self.tokenizer, and self.inputs on the runner instance.

Internal Behavior

The method performs the following steps in order:

1. Tokenizer loading (once per model ID): Calls AutoTokenizer.from_pretrained(model_id) and sets eos_token = pad_token for open-ended generation.
2. Input preparation: Tokenizes DEFAULT_PROMPT (a multi-paragraph text about the French Revolution) with truncation to config.sequence_length, replicating across config.batch_size sequences, and moves tensors to config.device. Sets use_cache=True.
3. Generation config: Creates a GenerationConfig with do_sample=False, max_new_tokens, and optionally compile_config and cache_implementation="static".
4. Model loading: Calls AutoModelForCausalLM.from_pretrained with dtype=torch.bfloat16, the specified attn_implementation, use_kernels flag, and device_map.
5. Evaluation mode: Calls self.model.eval().

Usage Examples

Basic Usage

import logging
from benchmark_v2.framework.benchmark_runner import BenchmarkRunner
from benchmark_v2.framework.benchmark_config import BenchmarkConfig

logger = logging.getLogger("benchmark")
runner = BenchmarkRunner(logger=logger, output_dir="./results")

config = BenchmarkConfig(
    attn_implementation="flash_attention_2",
    batch_size=1,
    sequence_length=128,
    num_tokens_to_generate=128,
)

# Load model and prepare inputs
runner.setup_benchmark("meta-llama/Llama-3-8B", config)

# runner.model, runner.tokenizer, and runner.inputs are now set
print(type(runner.model))  # AutoModelForCausalLM

Multiple Configurations for Same Model

# Tokenizer is loaded only once; model is reloaded for each config
config_eager = BenchmarkConfig(attn_implementation="eager")
config_sdpa = BenchmarkConfig(attn_implementation="sdpa")

runner.setup_benchmark("meta-llama/Llama-3-8B", config_eager)
# ... run benchmark ...
runner.cleanup()

runner.setup_benchmark("meta-llama/Llama-3-8B", config_sdpa)
# ... run benchmark ... (tokenizer reused, model reloaded)

Related Pages

Implements Principle

• Principle:Huggingface_Transformers_Benchmark_Model_Loading