Implementation:NVIDIA NeMo Aligner InferenceMetricsHandler
| Implementation Details | |
|---|---|
| Name | InferenceMetricsHandler |
| Type | API Doc |
| Module | nemo_aligner.metrics |
| Repository | NeMo Aligner |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
A unified wrapper for managing multiple metrics objects during inference and generation steps in NeMo Aligner training pipelines.
Description
The InferenceMetricsHandler class provides a single interface for orchestrating update, compute, and reset operations across a collection of registered metric objects. It is instantiated from an optional Hydra DictConfig that describes which metrics to create; if the config is None, all methods gracefully become no-ops and compute returns an empty dictionary. Internally, it delegates to hydra.utils.instantiate to build the metrics dictionary, making the set of tracked metrics fully config-driven and extensible without code changes.
The class follows the standard update / compute / reset lifecycle pattern: call update with each batch and its generation output during validation, call compute at the end of the validation run to retrieve finalized metric values, and call reset to clear internal state before the next run.
Usage
Used during validation and inference loops in NeMo Aligner trainers (e.g., PPO, REINFORCE, SFT) to collect and aggregate custom metrics over generated outputs. The handler is typically created once during trainer initialization and then invoked on every validation step.
Code Reference
Source Location
- Repository: NeMo Aligner
- File:
nemo_aligner/metrics/common.py(L23-54)
Signature
class InferenceMetricsHandler:
"""A wrapper around metrics objects that will call update/compute/reset on all registered metrics.
If metrics_config is None, then all methods become no-ops and compute will return an empty dict.
"""
def __init__(self, metrics_config: Optional[DictConfig]):
...
def has_metrics(self) -> bool:
"""Returns True if there are metrics to compute."""
...
def update(self, batch: Dict, generation_output: Dict):
"""Calling .update on all metrics."""
...
def compute(self) -> Dict[str, float]:
"""Returns a dictionary with finalized metric values."""
...
def reset(self):
"""Will reset state of all metrics to prepare for the next validation run."""
...
Import
from nemo_aligner.metrics.common import InferenceMetricsHandler
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| metrics_config | Optional[DictConfig] | Yes | Hydra-compatible config describing metrics to instantiate; None results in a no-op handler with no metrics
|
update() parameters:
| Name | Type | Required | Description |
|---|---|---|---|
| batch | Dict | Yes | A batch dictionary from the validation dataloader |
| generation_output | Dict | Yes | Output dictionary from model.generate containing generated tokens and related data
|
Outputs
| Name | Type | Description |
|---|---|---|
| has_metrics() | bool | True if one or more metrics are registered, False otherwise
|
| compute() | Dict[str, float] | Dictionary mapping metric names to their finalized scalar values; empty dict if no metrics are registered |
Usage Examples
from omegaconf import DictConfig
from nemo_aligner.metrics.common import InferenceMetricsHandler
# Initialize from Hydra config
metrics_cfg = DictConfig({"rouge": {"_target_": "my_metrics.RougeMetric"}})
handler = InferenceMetricsHandler(metrics_config=metrics_cfg)
# During validation loop
for batch in val_dataloader:
generation_output = model.generate(batch)
handler.update(batch, generation_output)
# After validation
results = handler.compute() # {"rouge": 0.85}
handler.reset()
# No-op handler (no metrics configured)
noop_handler = InferenceMetricsHandler(metrics_config=None)
assert not noop_handler.has_metrics()
assert noop_handler.compute() == {}