Principle:Huggingface Transformers Benchmark Statistics
| Knowledge Sources | |
|---|---|
| Domains | Benchmarking, Performance, Statistics |
| Last Updated | 2026-02-13 00:00 GMT |
Overview
Benchmark statistics computes summary descriptive statistics (mean, standard deviation, minimum, median, maximum, and 95th percentile) from raw measurement samples to characterize the central tendency, spread, and tail behavior of inference performance.
Description
Raw benchmark measurements are lists of per-iteration values (e.g., end-to-end latency in seconds). To draw meaningful conclusions, these raw samples must be summarized into descriptive statistics that capture different aspects of the distribution:
- Mean (avg): The arithmetic average of all measurements. Provides a single representative value for the typical performance, but is sensitive to outliers.
- Standard deviation (std): Measures the spread of measurements around the mean. A high standard deviation indicates unstable performance, which may be caused by thermal throttling, background processes, or non-deterministic GPU scheduling.
- Minimum (min): The best-case measurement. Useful for understanding the achievable peak performance under ideal conditions.
- Median (med): The 50th percentile. More robust to outliers than the mean and often a better representative of "typical" performance in skewed distributions.
- Maximum (max): The worst-case measurement. Important for understanding tail latency in production scenarios.
- 95th percentile (p95): The value below which 95% of measurements fall. This is the standard service-level objective (SLO) metric for latency-sensitive applications, capturing near-worst-case behavior while excluding extreme outliers.
The framework applies these statistics to multiple metric types within a BenchmarkResult:
- End-to-end latency
- Time to first token
- Inter-token latency
- Throughput (tokens per second, derived from e2e latency and total generated tokens)
Results are formatted with human-readable duration units (ns, us, ms, s, min, hr) and aligned for tabular display.
Usage
Use benchmark statistics when you need to:
- Summarize raw measurement samples into a concise performance profile.
- Compare two benchmark configurations by their mean, median, or p95 latency.
- Assess measurement stability by examining the standard deviation relative to the mean.
- Report performance metrics in a format suitable for dashboards, reports, or automated regression detection.
Theoretical Basis
The choice of six statistics reflects established practices in performance benchmarking and descriptive statistics:
- Central tendency: Both mean and median are reported because inference latency distributions are often right-skewed (occasional slow iterations due to GPU scheduling or thermal events). The median is robust to these outliers while the mean captures their aggregate impact.
- Dispersion: Standard deviation quantifies measurement noise. For a benchmark to be reliable, the coefficient of variation (std/mean) should be small (typically below 5-10%). High variance signals the need for more warmup iterations, more measurement iterations, or better environmental control (e.g., disabling frequency scaling).
- Percentiles: The 95th percentile is the industry-standard tail latency metric. It answers the question: "What latency will 95% of requests experience or better?" This is more actionable than the maximum, which can be dominated by a single extreme outlier, and more informative than the mean for latency-sensitive services.
- Min and max as bounds: These provide the full range of observed behavior. The gap between min and max indicates the total variability window. A large min-max range relative to the median suggests measurement instability.
- NumPy implementation: The framework uses
numpy.mean,numpy.std,numpy.min,numpy.median,numpy.max, andnumpy.percentilefor computation. These are numerically stable implementations that handle edge cases (empty arrays default to 0).