Principle:Openai Evals Completion Function Protocol
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, Software_Architecture |
| Last Updated | 2026-02-14 10:00 GMT |
Overview
A runtime-checkable protocol that defines the minimal interface any model integration must satisfy to be usable in the evaluation framework.
Description
The Completion Function Protocol defines a two-part contract: CompletionFn (a callable that takes a prompt and returns a result) and CompletionResult (an abstract class providing get_completions() -> list[str]). This protocol-based design allows any model — OpenAI, Anthropic, local models, or custom implementations — to participate in evaluations without inheritance from a specific base class. The @runtime_checkable decorator enables isinstance checks at runtime. The protocol is the original model abstraction in evals; the newer Solver interface provides a more stateful alternative.
Usage
Implement the CompletionFn protocol when integrating a new model or API with the evaluation framework. This is the minimal interface required for a model to work with oaieval.
Theoretical Basis
The protocol follows Python's structural subtyping pattern:
# The protocol contract (pseudocode)
class CompletionFn(Protocol):
def __call__(self, prompt, **kwargs) -> CompletionResult: ...
class CompletionResult(ABC):
def get_completions(self) -> list[str]: ...
Key design principles:
- Structural typing — No inheritance required; any object with the right method signatures satisfies the protocol
- Prompt flexibility — Accepts both raw strings and chat message lists
- Batch returns — get_completions() returns a list, supporting multi-sample generation
- Extensibility — Kwargs allow passing model-specific parameters (temperature, max_tokens, etc.)