Principle:Openai Evals Solver Implementation
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, Software_Architecture |
| Last Updated | 2026-02-14 10:00 GMT |
Overview
A stateful model integration pattern that supports multi-turn conversations, tool use, and composable behavior through a solver abstraction.
Description
Solver Implementation is the newer model integration pattern in the evals framework, designed for evaluations requiring stateful interactions. Unlike the simpler CompletionFn protocol (single prompt in, text out), Solver accepts a TaskState object containing conversation history and task description, and returns a SolverResult with output and metadata. Solvers support postprocessing pipelines, deep copying for per-sample isolation, and composition via NestedSolver (enabling chain-of-thought, few-shot prompting, and majority voting patterns). Solvers also inherit from CompletionFn for backward compatibility.
Usage
Implement a Solver when the evaluation requires multi-turn conversations, tool use, memory, or when you want to compose behaviors (e.g., chain-of-thought + few-shot). Use CompletionFn for simpler single-turn evaluations.
Theoretical Basis
The Solver pattern provides:
- TaskState — Carries task_description, messages (conversation history), and current_state
- SolverResult — Contains output string plus arbitrary metadata
- Postprocessors — Pipeline of output transformations applied after _solve
- NestedSolver — Composition pattern where a solver delegates to sub-solvers
- Per-sample copies — Each sample gets a fresh solver copy to prevent state leakage
Built-in solver compositions:
- CoTSolver — Adds chain-of-thought reasoning
- FewShotSolver — Prepends example interactions
- HHHSolver — Adds safety alignment prompts
- SelfConsistencySolver — Majority vote across multiple solvers