Implementation:Openai Evals Eval YAML Registration

Knowledge Sources	OpenAI Evals
Domains	Evaluation, Configuration
Last Updated	2026-02-14 10:00 GMT

Overview

Concrete pattern for registering evaluations via YAML configuration files consumed by the evals Registry system.

Description

Eval YAML Registration defines the pattern for adding new evaluations to the framework. YAML files in evals/registry/evals/ are loaded by Registry._load_registry which scans all .yaml files, parses each name-spec pair, validates reserved keywords, and stores them in a flat dictionary. The EvalSpec dataclass captures the parsed result with fields: cls (class path), args (constructor kwargs), registry_path, key, and group. Registration also supports the class keyword (automatically renamed to cls) and aliasing via string values or id fields.

Usage

Create a YAML file in evals/registry/evals/ following the two-level convention (base eval + versioned split). The eval becomes immediately discoverable by oaieval and Registry.get_eval().

Code Reference

Source Location

• Repository: openai/evals
• File: evals/registry.py (lines 287-310 for _load_registry), evals/base.py (lines 50-60 for EvalSpec)

Signature

@dataclass
class EvalSpec:
    cls: str
    registry_path: Path
    args: Optional[Dict[str, Any]] = None
    key: Optional[str] = None
    group: Optional[str] = None

YAML Schema

# Base eval entry
<eval-name>:
  id: <eval-name>.<split>.<version>    # Points to default split
  metrics: [<metric1>, <metric2>]       # Optional metadata
  description: "Human-readable description"

# Versioned split entry
<eval-name>.<split>.<version>:
  class: <fully.qualified.ClassName>    # Python class path
  args:                                 # Constructor kwargs
    samples_jsonl: <path/to/data.jsonl>
    max_tokens: <int>
    # ... additional class-specific args

I/O Contract

Inputs

Name	Type	Required	Description
YAML file	.yaml file	Yes	File in evals/registry/evals/ with eval definitions
cls/class	str	Yes	Fully qualified Python class path
args	dict	No	Constructor keyword arguments for the eval class

Outputs

Name	Type	Description
Registered eval	EvalSpec	Eval discoverable via Registry.get_eval(name)

Usage Examples

Register a Match Eval

# File: evals/registry/evals/my_task.yaml
my-task:
  id: my-task.dev.v0
  metrics: [accuracy]

my-task.dev.v0:
  class: evals.elsuite.basic.match.Match
  args:
    samples_jsonl: my_data/questions.jsonl
    max_tokens: 200

Register a Custom Eval

# File: evals/registry/evals/custom.yaml
my-custom-eval:
  id: my-custom-eval.dev.v0
  metrics: [accuracy, f1]
  description: "Custom evaluation with specialized scoring"

my-custom-eval.dev.v0:
  class: my_package.my_module.MyCustomEval
  args:
    samples_jsonl: my_data/custom.jsonl
    threshold: 0.8

Register a Model-Graded Eval

# File: evals/registry/evals/graded.yaml
my-graded-eval:
  id: my-graded-eval.dev.v0
  metrics: [score]

my-graded-eval.dev.v0:
  class: evals.elsuite.modelgraded.classify.ModelBasedClassify
  args:
    samples_jsonl: my_data/graded.jsonl
    modelgraded_spec: fact
    eval_type: cot_classify

Running After Registration

# The eval is now accessible via CLI
oaieval gpt-3.5-turbo my-task
oaieval gpt-4 my-custom-eval --max_samples 50

Related Pages

Implements Principle

• Principle:Openai_Evals_Eval_Registration

Uses Heuristic

• Heuristic:Openai_Evals_Chat_Format_Recommendation