Implementation:Openai Evals Registry Get Eval

Knowledge Sources	OpenAI Evals
Domains	Evaluation, Configuration
Last Updated	2026-02-14 10:00 GMT

Overview

Concrete tool for looking up an evaluation specification by name from the YAML registry provided by the evals Registry class.

Description

The Registry.get_eval method performs a dictionary lookup in the lazily-loaded eval registry, resolving aliases and returning an EvalSpec dataclass. The registry is loaded on first access via the _evals cached property, which scans all YAML files in evals/registry/evals/ directories across all configured registry paths. If the name is not found, it logs a warning with the closest matching names using difflib.get_close_matches.

Usage

Use this method when you need to look up an eval by name before running it. Called internally by oaieval.run() and can be used programmatically for eval discovery.

Code Reference

Source Location

• Repository: openai/evals
• File: evals/registry.py (line 210-211)

Signature

class Registry:
    def get_eval(self, name: str) -> Optional[EvalSpec]:
        """
        Look up an eval by name, resolving aliases.

        Args:
            name: Eval name (e.g. "test-match", "my-eval.dev.v0").

        Returns:
            EvalSpec if found, None otherwise.
        """

Import

from evals.registry import Registry
from evals.base import EvalSpec

I/O Contract

Inputs

Name	Type	Required	Description
name	str	Yes	Eval name matching a YAML entry in evals/registry/evals/

Outputs

Name	Type	Description
return value	Optional[EvalSpec]	EvalSpec with cls, args, registry_path, key, group fields; None if not found

Usage Examples

Look Up a Built-in Eval

from evals.registry import Registry

registry = Registry()
eval_spec = registry.get_eval("test-match")

if eval_spec:
    print(f"Class: {eval_spec.cls}")        # "evals.elsuite.basic.match.Match"
    print(f"Args: {eval_spec.args}")          # {"samples_jsonl": "test/..."}
    print(f"Registry: {eval_spec.registry_path}")

Check if Eval Exists

from evals.registry import Registry

registry = Registry()
spec = registry.get_eval("nonexistent-eval")
# Logs warning: "eval 'nonexistent-eval' not found. Closest matches: [...]"
assert spec is None

Related Pages

Implements Principle

• Principle:Openai_Evals_Eval_Resolution