Implementation:Huggingface Transformers Benchmark V1 Runner
| Knowledge Sources | |
|---|---|
| Domains | Benchmarking, Performance_Testing |
| Last Updated | 2026-02-13 20:00 GMT |
Overview
Concrete tool for running benchmarks using the optimum-benchmark library across one or more git commits and aggregating results.
Description
The benchmark.py module orchestrates the v1 benchmark system. It parses CLI arguments for config paths, commit lists, metrics, and Hub upload settings. For each specified commit, it checks out that commit via a checkout_commit context manager, then invokes optimum_benchmark_wrapper.main with Hydra-style config arguments for each model. After runs complete, it collects benchmark_report.json files via summarize(), extracts selected metrics, and produces per-run and combined JSON summaries. combine_summaries() aggregates results by model, config, and commit. Results can be uploaded to a HuggingFace dataset repo via HfApi.upload_folder.
Usage
Use this script to run performance benchmarks across multiple git commits to track performance regressions. It is typically invoked by CI pipelines or manually by maintainers for performance comparisons.
Code Reference
Source Location
- Repository: Huggingface_Transformers
- File: benchmark/benchmark.py
- Lines: 1-324
Signature
@contextlib.contextmanager
def checkout_commit(repo: git.Repo, commit_id: str):
"""Context manager that checks out a specific commit and restores the original state on exit."""
def summarize(folder: str, metrics: List[str]) -> Dict:
"""Collect benchmark_report.json files and extract selected metrics."""
def combine_summaries(summaries: List[Dict]) -> Dict:
"""Aggregate results by model, config, and commit."""
Import
python benchmark/benchmark.py --config-dir configs/ --config-name my_config \
--commit_id abc123 def456 --metrics latency memory
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| --config-dir | str | Yes | Directory containing benchmark config files |
| --config-name | str | Yes | Name of the benchmark configuration |
| --commit_id | List[str] | No | Git commit SHAs to benchmark |
| --metrics | List[str] | No | Metrics to extract from results |
| --repo_id | str | No | HuggingFace Hub repo for uploading results |
Outputs
| Name | Type | Description |
|---|---|---|
| benchmark_report.json | JSON files | Per-run benchmark results |
| summary.json | JSON file | Aggregated summary of all runs |
Usage Examples
Running Benchmarks Across Commits
# Run benchmarks for two commits
python benchmark/benchmark.py \
--config-dir benchmark/configs/ \
--config-name default \
--commit_id abc123 def456 \
--metrics latency throughput memory \
--repo_id huggingface/transformers-benchmarks