Implementation:Huggingface Transformers Save Results

Knowledge Sources	Transformers
Domains	Benchmarking, Performance, Data Persistence
Last Updated	2026-02-13 00:00 GMT

Overview

Concrete tool for serializing benchmark results to JSON files and optionally pushing them to a Hugging Face Hub dataset, provided by the HuggingFace Transformers benchmark framework.

Description

BenchmarkRunner.save_results serializes all benchmark results (metadata, measurements, and configuration) to a timestamped JSON file in a model-specific subdirectory. The JSON output uses a compact formatting for numeric arrays (collapsing them to single lines) to balance human readability with file size. Each result entry is keyed by the configuration's SHA-256 hash. The method is called after each benchmark configuration completes, providing incremental persistence and crash resilience.

BenchmarkRunner.push_results_to_hub uploads results to a Hugging Face Hub dataset repository in JSONL format. It performs two uploads: one with summarized results (omitting raw timestamps and GPU metrics for compact storage) and one with full results (including all raw data). Authentication is handled via the PUSH_TO_HUB_TOKEN environment variable.

Usage

save_results is called automatically by run_benchmarks after each configuration completes. push_results_to_hub is called explicitly when results need to be shared via the Hugging Face Hub. Both methods can also be called manually for custom persistence workflows.

Code Reference

Source Location

• Repository: transformers
• File: benchmark_v2/framework/benchmark_runner.py (lines 402-470)

Signature

def save_results(
    self,
    model_name: str,
    results: dict,
    timestamp: str = "",
    summarized: bool = True,
) -> str:
    ...

def push_results_to_hub(
    self,
    dataset_id: str,
    results: dict[Any, Any],
    timestamp: str,
) -> None:
    ...

Import

from benchmark_v2.framework.benchmark_runner import BenchmarkRunner

I/O Contract

Inputs (save_results)

Name	Type	Required	Description
model_name	str	Yes	Model identifier (e.g., "meta-llama/Llama-3-8B"). Slashes are replaced with underscores for directory naming.
results	dict	Yes	Dictionary keyed by configuration hash, each value containing "metadata" (BenchmarkMetadata), "measurements" (BenchmarkResult), and "config" (BenchmarkConfig).
timestamp	str	No (default: "")	Timestamp string for the filename. Auto-generated in YYYYMMDD_HHMMSS format if empty.
summarized	bool	No (default: True)	Whether to omit raw timestamps and GPU metrics from the output.

Outputs (save_results)

Name	Type	Description
filepath	str	Absolute path to the saved JSON file.

Inputs (push_results_to_hub)

Name	Type	Required	Description
dataset_id	str	Yes	Hugging Face Hub dataset repository ID (e.g., "org/benchmark-results").
results	dict[Any, Any]	Yes	Same results dictionary as save_results.
timestamp	str	Yes	Timestamp string for organizing uploads.

Outputs (push_results_to_hub)

Name	Type	Description
(side effect)	N/A	Uploads two JSONL files to the Hub: summarized_results/benchmark_run_{timestamp}.jsonl and full_results/benchmark_run_{timestamp}.jsonl.

File Structure

The local output follows this directory structure:

{output_dir}/
  {model_name}/
    {model_name}_benchmark_{timestamp}.json

Each JSON file contains a dictionary where keys are configuration SHA-256 hashes and values have three sections:

{
    "a1b2c3d4...": {
        "metadata": {
            "model_id": "meta-llama/Llama-3-8B",
            "timestamp": "2024-01-15T14:30:22+00:00",
            "branch_name": "main",
            "commit_id": "abc123",
            "commit_message": "...",
            "hardware_info": {"gpu_name": "NVIDIA H100", ...},
            "success": true
        },
        "measurements": {
            "e2e_latency": [0.45, 0.43, ...],
            "time_to_first_token": [0.012, 0.011, ...],
            "inter_token_latency": [0.0034, 0.0033, ...],
            "shape_and_decoded_outputs": ["(1, 128) | The French...", ...],
            "gpu_metrics": null,
            "timestamps": null
        },
        "config": {
            "name": "w5_i20-monitored-b1_s128_n128-eager-uncompiled-...",
            "warmup_iterations": 5,
            ...
        }
    }
}

Usage Examples

Basic Usage

import logging
from benchmark_v2.framework.benchmark_runner import BenchmarkRunner
from benchmark_v2.framework.benchmark_config import get_config_by_level

logger = logging.getLogger("benchmark")
runner = BenchmarkRunner(logger=logger, output_dir="./results")

configs = get_config_by_level(level=1)
timestamp, all_results = runner.run_benchmarks("meta-llama/Llama-3-8B", configs)
# save_results is called automatically after each config completes

# Results are saved to: ./results/meta-llama_Llama-3-8B/meta-llama_Llama-3-8B_benchmark_{timestamp}.json

Manual Save

# Save results manually with a custom timestamp
filepath = runner.save_results(
    model_name="meta-llama/Llama-3-8B",
    results=all_results,
    timestamp="20240115_143022",
    summarized=False,  # Include raw timestamps and GPU metrics
)
print(f"Saved to: {filepath}")

Push to Hub

import os
os.environ["PUSH_TO_HUB_TOKEN"] = "hf_..."

runner.push_results_to_hub(
    dataset_id="huggingface/benchmark-results",
    results=all_results,
    timestamp=timestamp,
)
# Uploads both summarized_results/ and full_results/ JSONL files

Related Pages

Implements Principle

• Principle:Huggingface_Transformers_Benchmark_Results_Persistence