Principle:Openai Evals Completion Function Resolution
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, Model_Integration |
| Last Updated | 2026-02-14 10:00 GMT |
Overview
A resolution mechanism that maps a string identifier to an executable model completion function through a multi-strategy lookup process.
Description
Completion Function Resolution is the process of converting a user-provided name string (such as a model ID or registry key) into an instantiated CompletionFn object ready for evaluation. The resolution follows a priority chain: first checking for the special "dummy" identifier, then attempting to match a known OpenAI model name (distinguishing between chat and completion models), and finally falling back to a YAML registry lookup for custom completion functions or solvers. This multi-strategy approach allows the same CLI interface to work with built-in OpenAI models, custom completion functions, and solver-based implementations.
Usage
Use this principle whenever a model or completion function needs to be identified and instantiated from a string name. This is the entry point of every evaluation: the first positional argument to oaieval is resolved through this mechanism.
Theoretical Basis
The resolution follows a cascading lookup pattern:
# Pseudocode for resolution strategy
def resolve(name):
if name == "dummy":
return DummyCompletionFn()
if is_known_chat_model(name):
return OpenAIChatCompletionFn(model=name)
if is_known_api_model(name):
return OpenAICompletionFn(model=name)
spec = registry.get_completion_fn(name) or registry.get_solver(name)
if spec:
return instantiate_from_spec(spec)
raise ValueError("not found")
Key design decisions:
- Chat vs completion model distinction based on model name prefix matching
- Context window size (n_ctx) automatically determined from model name
- Registry lookup merges completion_fns and solvers namespaces
- Additional kwargs can be passed through from CLI --completion_args