Principle:Openai Evals YAML Registry Configuration
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, Configuration |
| Last Updated | 2026-02-14 10:00 GMT |
Overview
A declarative YAML configuration pattern for registering completion functions and solvers with the evaluation framework.
Description
YAML Registry Configuration is the mechanism for registering custom CompletionFn and Solver implementations so they can be referenced by name in the oaieval CLI. YAML files in evals/registry/completion_fns/ and evals/registry/solvers/ define name-to-class mappings with constructor arguments. The CompletionFnSpec dataclass captures each registration with cls (fully-qualified class path), args (constructor kwargs), and registry metadata. This declarative approach avoids code changes when adding or reconfiguring model integrations.
Usage
Register a completion function or solver after implementing it to make it available via oaieval <name> <eval>. Also used for configuring existing solver compositions with different parameters.
Theoretical Basis
YAML registration structure:
# Completion function registration
my-custom-fn:
class: my_package.my_module:MyCompletionFn
args:
model: "my-model"
api_base: "https://my-api.com/v1"
# Solver registration
my-solver:
class: my_package.solver:MySolver
args:
model_name: "my-model"
postprocessors:
- "evals.solvers.postprocessors.strip:Strip"
Registry lookup merges: When resolving a name, the registry checks both completion_fns and solvers namespaces (_completion_fns | _solvers).