Implementation:Openai Evals CompletionFnSpec Registration

Knowledge Sources	OpenAI Evals
Domains	Evaluation, Configuration
Last Updated	2026-02-14 10:00 GMT

Overview

Concrete pattern for registering completion functions and solvers via YAML configuration consumed by the evals Registry system.

Description

YAML files in evals/registry/completion_fns/ and evals/registry/solvers/ are loaded by Registry._load_registry into the _completion_fns and _solvers cached properties. Each entry is parsed into a CompletionFnSpec dataclass. When Registry.make_completion_fn receives a name not matching a known model, it looks up the spec and uses make_object to instantiate the class with the provided args.

Usage

Create a YAML file in the appropriate registry directory to register a completion function or solver. The name becomes immediately usable with oaieval.

Code Reference

Source Location

• Repository: openai/evals
• File: evals/registry.py (lines 287-318 for _load_registry and cached properties), evals/base.py (lines 17-26 for CompletionFnSpec)

Signature

@dataclass
class CompletionFnSpec:
    cls: str
    args: Optional[Dict[str, Any]] = None
    key: Optional[str] = None
    group: Optional[str] = None

YAML Schema

<fn-name>:
  class: <fully.qualified.module:ClassName>
  args:
    <key1>: <value1>
    <key2>: <value2>

I/O Contract

Inputs

Name	Type	Required	Description
YAML file	.yaml file	Yes	File in evals/registry/completion_fns/ or evals/registry/solvers/
cls/class	str	Yes	Fully qualified class path (module:ClassName format)
args	dict	No	Constructor keyword arguments

Outputs

Name	Type	Description
Registered fn/solver	CompletionFnSpec	Discoverable via Registry.make_completion_fn(name)

Usage Examples

Register a Custom CompletionFn

# File: evals/registry/completion_fns/custom.yaml
my-custom-fn:
  class: my_package.completions:MyCompletionFn
  args:
    model: "my-model-v1"
    api_base: "https://my-api.com/v1"
    temperature: 0.0

Register a Solver

# File: evals/registry/solvers/my_solver.yaml
my-cot-solver:
  class: evals.solvers.nested.cot_solver:CoTSolver
  args:
    cot_solver:
      class: evals.solvers.providers.openai.openai_solver:OpenAISolver
      args:
        completion_fn_options:
          model: gpt-4
          max_tokens: 1024
    extract_solver:
      class: evals.solvers.providers.openai.openai_solver:OpenAISolver
      args:
        completion_fn_options:
          model: gpt-3.5-turbo
          max_tokens: 256

Running with Custom Registration

# Use registered name directly
oaieval my-custom-fn test-match
oaieval my-cot-solver test-match

# Add custom registry path
oaieval my-fn test-match --registry_path ./my_registry

Related Pages

Implements Principle

• Principle:Openai_Evals_YAML_Registry_Configuration