Implementation:Huggingface Transformers Grafana Benchmark Dashboard

Knowledge Sources	Huggingface_Transformers
Domains	Benchmarking, Observability
Last Updated	2026-02-13 20:00 GMT

Overview

Concrete configuration artifact for visualizing Transformers benchmark results in Grafana, including model timing metrics and hardware utilization over time.

Description

The grafana_dashboard.json file defines a complete Grafana dashboard for the v1 benchmark infrastructure. It contains panels for: a table of recent benchmark commits, bar charts for first/second eager forward pass times, time-to-first/second/third token, mean time-to-next-token, and compiled generate timing charts. A collapsed "Usage metrics" section includes timeseries panels for CPU utilization, GPU utilization, memory usage, and GPU memory usage. All panels query a PostgreSQL datasource joining benchmarks, model_measurements, and device_measurements tables. Template variables allow filtering by branch, GPU name, and number of recent commits.

Usage

Deploy this dashboard to a Grafana instance connected to the benchmark PostgreSQL database. It is used at transformers-benchmarks.hf.co to monitor performance regressions across commits.

Code Reference

Source Location

• Repository: Huggingface_Transformers
• File: benchmark/grafana_dashboard.json
• Lines: 1-2375

Signature

{
  "title": "Transformers benchmarks",
  "uid": "...",
  "panels": [
    {"type": "table", "title": "Commits"},
    {"type": "barchart", "title": "First eager forward"},
    {"type": "timeseries", "title": "GPU utilization"}
  ],
  "templating": {
    "list": [
      {"name": "branch", "type": "query"},
      {"name": "gpu_name", "type": "query"},
      {"name": "num_recent_commits", "type": "custom"}
    ]
  }
}

Import

# Import via Grafana API or UI
grafana-cli dashboards import benchmark/grafana_dashboard.json

I/O Contract

Inputs

Name	Type	Required	Description
PostgreSQL datasource	Database	Yes	Database containing benchmarks, model_measurements, device_measurements tables
branch	Template variable	No	Filter by git branch
gpu_name	Template variable	No	Filter by GPU type

Outputs

Name	Type	Description
Dashboard panels	Visual	Charts and tables rendered in Grafana

Usage Examples

Deploying the Dashboard

# 1. Configure PostgreSQL datasource (see grafana_datasource.yaml)
# 2. Import the dashboard JSON via Grafana UI:
#    Dashboards > Import > Upload JSON file > benchmark/grafana_dashboard.json
# 3. Select the PostgreSQL datasource when prompted

Related Pages

• Principle:Huggingface_Transformers_Benchmark_Orchestration_V1