Principle:Openai Evals Eval Orchestration
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, Orchestration |
| Last Updated | 2026-02-14 10:00 GMT |
Overview
A pipeline orchestration pattern that coordinates the full lifecycle of a single evaluation run from argument parsing through result recording.
Description
Eval Orchestration is the top-level coordination process that ties together all evaluation components into a single execution pipeline. It resolves the completion function and eval specification from their string names, constructs a RunSpec for tracking, initializes a recorder backend, instantiates the eval class, executes the evaluation, collects token usage statistics, and records the final report. This orchestration layer is what the oaieval CLI delegates to after parsing command-line arguments.
Usage
Use this orchestration pattern whenever a complete end-to-end evaluation run is needed. This is the primary entry point for all single-eval execution, whether invoked via CLI or programmatically.
Theoretical Basis
The orchestration follows a linear pipeline:
- Parse arguments — CLI flags to structured OaiEvalArguments
- Resolve completion function — String name to CompletionFn instance
- Resolve eval — String name to EvalSpec dataclass
- Build recorder — Select and configure recording backend
- Instantiate eval — Create Eval class from spec with completion functions
- Execute — Call Eval.run(recorder) which internally calls eval_all_samples
- Record results — Token usage appended, final report recorded
- Return run_id — Unique identifier for this evaluation run
# Pseudocode for orchestration
def orchestrate(model_name, eval_name, args):
completion_fn = registry.make_completion_fn(model_name)
eval_spec = registry.get_eval(eval_name)
recorder = build_recorder(args, run_spec)
eval_instance = eval_spec.cls(completion_fns=[completion_fn], ...)
result = eval_instance.run(recorder)
recorder.record_final_report(result)
return run_spec.run_id