Principle:Openai Evals Eval Metrics
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, Statistics |
| Last Updated | 2026-02-14 10:00 GMT |
Overview
A collection of standard statistical metrics for quantifying evaluation results from match events.
Description
Eval Metrics provides the standard statistical functions used to aggregate per-sample evaluation results into summary metrics. The core metric is accuracy (fraction of correct matches), supplemented by bootstrap standard deviation (uncertainty estimate via resampling), confusion matrix construction, and derived metrics including precision, recall, F-score, and Matthew's correlation coefficient. These metrics operate on Event objects recorded during evaluation and provide the final numbers reported to users.
Usage
Use eval metrics in the run method of custom Eval classes to aggregate recorded events into summary statistics. The standard pattern is to call recorder.get_events("match") and pass the events to metric functions.
Theoretical Basis
Core metrics:
- Accuracy = correct / total (fraction of exact matches)
- Bootstrap Std = standard deviation of accuracy estimates from resampled subsets (default 1000 resamples of half the data)
- Confusion Matrix = N×(N+1) matrix counting (expected, picked) pairs, with an extra column for unrecognized picks
- Precision = TP / (TP + FP) for a given class
- Recall = TP / (TP + FN) for a given class
- F-score = (1 + β²) × (precision × recall) / (β² × precision + recall)