Principle:Openai Evals Eval Template Selection
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, NLP |
| Last Updated | 2026-02-14 10:00 GMT |
Overview
A selection process for choosing an appropriate built-in evaluation template based on the desired matching strategy between model outputs and expected answers.
Description
Eval Template Selection involves choosing from the built-in eval classes (Match, Includes, FuzzyMatch) based on how strictly model outputs should be compared against expected answers. Match requires exact string equality, Includes checks if the expected answer appears as a substring, and FuzzyMatch uses normalized token-level F1 scoring. Each template extends the Eval base class and handles data loading, completion function invocation, and metric recording automatically. Selecting the right template avoids writing custom eval classes for common comparison patterns.
Usage
Select an eval template when creating a new evaluation that uses standard string comparison. Use Match for tasks with precise expected outputs (e.g., classification labels), Includes for tasks where the answer may appear within a longer response, and FuzzyMatch for tasks where approximate matching is acceptable (e.g., free-form text).
Theoretical Basis
Matching strategies:
- Exact Match — Binary: sampled output must exactly equal one of the expected strings
- Includes — Substring check: expected string must appear within the sampled output (optionally case-insensitive)
- Fuzzy Match — Token-level F1 score comparing normalized word tokens between sampled and expected strings
Each template records match events and computes accuracy with bootstrap standard deviation.