Principle:Huggingface Transformers Benchmark Results Persistence

Knowledge Sources	Transformers Docs Hugging Face Hub Datasets
Domains	Benchmarking, Performance, Data Persistence
Last Updated	2026-02-13 00:00 GMT

Overview

Benchmark results persistence serializes benchmark measurements, metadata, and configuration to JSON files on disk and optionally pushes them to a Hugging Face Hub dataset for centralized storage and historical tracking.

Description

Benchmark results are only useful if they survive the process that generated them. The HuggingFace Transformers benchmark framework implements a two-tier persistence strategy:

• Local JSON serialization: After each benchmark configuration completes (not just at the end of the full run), results are saved to a JSON file on disk. This provides crash resilience: if the process terminates unexpectedly during a long multi-configuration benchmark sweep, all previously completed configurations are preserved. The file structure is:
  • A model-specific subdirectory under the output directory (e.g., benchmark_results/meta-llama_Llama-3-8B/).
  • A timestamped JSON file (e.g., meta-llama_Llama-3-8B_benchmark_20240115_143022.json).
  • Each entry in the JSON is keyed by the configuration hash and contains three sections: metadata, measurements, and config.

• Hub dataset upload: Results can optionally be pushed to a Hugging Face Hub dataset repository. This supports:
  • Summarized results: Compact representations without raw timestamps, suitable for dashboards and trend analysis.
  • Full results: Complete data including all raw timestamps and GPU metrics, suitable for deep analysis and reproduction.
  • Both formats are uploaded as JSONL files organized by timestamp.

The serialization format handles special cases:

• Numeric arrays are compacted: A custom JSON formatter collapses arrays of numbers onto single lines to reduce file size while maintaining human readability.
• GPU metrics handling: If GPU monitoring was disabled or collected no samples, the metrics field is serialized as null rather than empty arrays.
• Summarized vs. full modes: The summarized flag controls whether raw timestamps and GPU metrics are included or omitted, trading completeness for file size.

Usage

Use benchmark results persistence when you need to:

• Save benchmark results for later analysis or comparison across commits and branches.
• Share benchmark data with team members via the Hugging Face Hub.
• Build historical performance dashboards that track regression across releases.
• Ensure that long-running benchmark sweeps do not lose data on failure.

Theoretical Basis

Results persistence follows principles from scientific data management:

• Reproducibility: Each saved result includes the full configuration, git commit ID, branch name, and hardware information. This allows any result to be reproduced by checking out the same commit, using the same hardware, and applying the same configuration.
• Incremental saving: Saving after each configuration (rather than at the end of the full run) follows the principle of write-ahead logging from database systems. This bounds the maximum data loss to a single configuration's worth of measurements, regardless of how many configurations remain.
• Schema versioning through content hashing: Using the configuration's SHA-256 hash as the result key enables deduplication. If the same configuration is run twice (e.g., due to a restart), the second run overwrites the first rather than creating duplicate entries.
• Two-level summarization: The full/summarized dichotomy balances the needs of different consumers. Automated dashboards need only summary statistics (small payloads, fast to load), while researchers diagnosing anomalies need raw per-iteration timestamps and GPU traces (large payloads, high fidelity).
• Hub integration: Centralizing results on the Hugging Face Hub leverages the existing infrastructure for dataset versioning, access control, and discovery. The JSONL format supports streaming reads and incremental appends.

Related Pages

Implemented By

• Implementation:Huggingface_Transformers_Save_Results