Implementation:Openai Evals Registry Get Eval Set

Knowledge Sources	OpenAI Evals
Domains	Evaluation, Configuration
Last Updated	2026-02-14 10:00 GMT

Overview

Concrete tool for looking up an eval set specification by name from the YAML registry provided by the evals Registry class.

Description

The Registry.get_eval_set method resolves an eval set name to an EvalSetSpec dataclass via the same alias-dereferencing mechanism used for evals. The eval set registry is lazily loaded from YAML files in evals/registry/eval_sets/ directories.

Usage

Use this method when looking up a batch of evals to run. Called internally by oaievalset.run().

Code Reference

Source Location

• Repository: openai/evals
• File: evals/registry.py (lines 213-214)

Signature

class Registry:
    def get_eval_set(self, name: str) -> Optional[EvalSetSpec]:
        """
        Look up an eval set by name.

        Args:
            name: Eval set name matching a YAML entry in evals/registry/eval_sets/.

        Returns:
            EvalSetSpec with evals list, or None if not found.
        """

Import

from evals.registry import Registry
from evals.base import EvalSetSpec

I/O Contract

Inputs

Name	Type	Required	Description
name	str	Yes	Eval set name (e.g. "test-basic", "test-modelgraded")

Outputs

Name	Type	Description
return value	Optional[EvalSetSpec]	EvalSetSpec with evals: Sequence[str], key, group; None if not found

Usage Examples

Look Up an Eval Set

from evals.registry import Registry

registry = Registry()
eval_set = registry.get_eval_set("test-basic")

if eval_set:
    print(f"Evals in set: {eval_set.evals}")
    for eval_name in eval_set.evals:
        spec = registry.get_eval(eval_name)
        print(f"  {eval_name}: {spec.cls if spec else 'not found'}")

Related Pages

Implements Principle

• Principle:Openai_Evals_Eval_Set_Resolution