Implementation:FMInference FlexLLMGen HELM Metric Evaluate

Knowledge Sources	FlexLLMGen HELM Benchmark
Domains	Evaluation, Metrics
Last Updated	2026-02-09 00:00 GMT

Overview

Wrapper documentation for HELM's metric evaluation pipeline as used by FlexLLMGen's benchmark integration.

Description

This Wrapper Doc covers the external HELM metric APIs. FlexLLMGen's helm_run.py creates metrics via create_metric(metric_spec) and evaluates them via metric.evaluate(scenario_state, tokenizer_service, eval_cache_path, parallelism). Results include Stat objects (aggregate metrics) and PerInstanceStats (per-example metrics), serialized to JSON files.

External Reference

• HELM Documentation

Code Reference

• Source: flexllmgen/apps/helm_run.py, Lines: 339-381
• Key HELM APIs:

# From helm.benchmark.runner
create_metric(metric_spec: MetricSpec) -> Metric

# Metric.evaluate
metric.evaluate(
    scenario_state: ScenarioState,
    metric_service: Any,  # tokenizer_service
    eval_cache_path: str,
    parallelism: int = 4
) -> MetricResult

• Import:

from helm.benchmark.runner import create_metric, MetricResult, Stat, PerInstanceStats

I/O Contract

Inputs

Parameter	Type	Required	Description
metric_specs	List[MetricSpec]	Yes	From RunSpec
scenario_state	ScenarioState	Yes	Completed with generation results
tokenizer_service	OptTokenizer	Yes	For token-level metrics
eval_cache_path	str	Yes	Cache directory
parallelism	int	No	Parallel evaluation threads default 4

Outputs

• MetricResult — with aggregated_stats: List[Stat] and per_instance_stats: List[PerInstanceStats]
• JSON files: run_spec.json, scenario.json, scenario_state.json, stats.json, per_instance_stats.json

Usage Examples

# Internal usage in run_entry():
from helm.benchmark.runner import create_metric

for metric_spec in run_spec.metric_specs:
    metric = create_metric(metric_spec)
    metric_result = metric.evaluate(
        scenario_state,
        metric_service=opt_tokenizer,
        eval_cache_path=eval_cache_path,
        parallelism=4
    )
    stats.extend(metric_result.aggregated_stats)
    per_instance_stats.extend(metric_result.per_instance_stats)

# Results saved as JSON
write(os.path.join(run_path, "stats.json"), [asdict_without_nones(s) for s in stats])

Related Pages

• Principle:FMInference_FlexLLMGen_HELM_Metric_Computation