Principle:FMInference FlexLLMGen HELM Metric Computation
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, Metrics |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A standardized evaluation methodology that computes scenario-specific metrics (accuracy, F1, calibration, fairness) on model generation results using HELM's metric framework.
Description
After model generation, HELM metrics evaluate quality by comparing generated text against ground truth references. The metric pipeline creates Metric objects from MetricSpecs (e.g., BasicMetrics for accuracy, TokensMetric for efficiency), runs metric.evaluate() which processes the completed ScenarioState, and produces MetricResult objects containing Stat aggregates and PerInstanceStats. Results are serialized to JSON for analysis and leaderboard submission.
Usage
Used automatically after generation in the HELM evaluation pipeline. Metrics are specified by the RunSpec and computed in parallel.
Theoretical Basis
HELM's evaluation framework standardizes LLM evaluation across accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency dimensions.