Principle:Sail sg LongSpec Metrics Collection
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, Benchmarking, Performance_Analysis |
| Last Updated | 2026-02-14 05:00 GMT |
Overview
Principle for measuring and reporting speculative decoding performance through token throughput, acceptance rate, and per-sample timing metrics.
Description
Metrics Collection quantifies the speedup achieved by speculative decoding compared to vanilla autoregressive generation. The key metrics are:
- Throughput (tokens/second): Total generated tokens divided by wall-clock time, the primary performance metric
- Acceptance rate: Ratio of accepted draft tokens to total draft tokens proposed, measuring draft model quality
- Per-sample timing: Individual sample generation time for variance analysis
The collection includes a warmup iteration (excluded from timing) to account for CUDA kernel compilation and memory allocation on the first inference pass.
Metrics are either printed to stdout (LongBench) or written to log files (AIME: ./long-bench_results/output_aime.txt).
Usage
Apply when benchmarking speculative decoding performance. All three generation methods (tree, sequential, vanilla) produce metrics, but acceptance rate is only available for speculative methods.
Theoretical Basis
Throughput calculation:
Failed to parse (syntax error): {\displaystyle \text{Throughput} = \frac{\text{total\_tokens}}{\text{elapsed\_time}} \text{ tokens/second} }
Acceptance rate:
Failed to parse (syntax error): {\displaystyle \text{Acceptance Rate} = \frac{\text{accepted\_tokens}}{\text{total\_draft\_tokens}} }
Speedup factor: