Principle:Openai Evals Multi Provider Solver Integration
| Knowledge Sources | |
|---|---|
| Domains | Evaluation, API Integration, Multi-Provider |
| Last Updated | 2026-02-14 10:00 GMT |
Overview
A principle that enables standardized model evaluation across multiple AI providers through a unified Solver interface, abstracting provider-specific API differences behind a common contract.
Description
Multi-provider solver integration addresses the practical challenge of evaluating and comparing models from different AI providers within a single, consistent evaluation framework. Each provider (OpenAI, Anthropic, Google, Together AI, and others) exposes models through distinct APIs with different message formats, authentication mechanisms, error handling patterns, and behavioral constraints. This principle defines a unified abstraction layer that allows evaluation code to remain provider-agnostic while each provider solver handles the specifics of its API.
The framework supports the following providers:
- Anthropic Claude (AnthropicSolver) -- Integration with Claude models via the Anthropic Messages API. Key considerations include role mapping (Anthropic requires alternating user/assistant messages and does not support a system role in the message list; system messages are passed separately), content block handling (Anthropic uses structured content blocks rather than plain strings), and safety filter responses (Anthropic may return refusal responses that must be handled gracefully).
- Google Gemini (GeminiSolver) -- Integration with Google's Gemini models. Provider-specific concerns include role name translation (Gemini uses "model" instead of "assistant"), message alternation enforcement (Gemini requires strict user/model alternation, requiring message merging when consecutive same-role messages occur), and safety settings configuration (Gemini provides fine-grained safety threshold controls).
- OpenAI Assistants API (OpenAIAssistantsSolver) -- Integration with OpenAI's Assistants API, which differs from the standard Chat Completions API by managing conversation threads server-side. This solver must handle thread creation and management, run polling (the Assistants API is asynchronous), and tool/function calling within the Assistants framework.
- Together AI (TogetherSolver) -- Integration with Together AI's inference platform for open-source models (LLaMA, Mixtral, etc.). This solver provides access to a wide range of community models through a unified API, enabling evaluation of open-source alternatives alongside proprietary models.
Each provider solver implements the same Solver interface, which defines:
- A method to receive a task in the evals message format and return a solver result.
- Standard error handling and retry logic with exponential backoff.
- API key management through environment variables or configuration.
- Token usage tracking for cost estimation.
Usage
Apply multi-provider solver integration when:
- You need to compare models from different providers on the same benchmark using identical evaluation conditions.
- You want to add support for a new provider without modifying any evaluation logic or benchmark code.
- You are running evaluations across proprietary and open-source models and need consistent scoring.
- You need to handle provider-specific edge cases (safety filters, rate limits, message format constraints) without polluting the evaluation code.
- You want to abstract away authentication and connection management from the evaluation pipeline.
Theoretical Basis
The multi-provider integration principle is grounded in the adapter pattern from software engineering, which converts the interface of one class into another interface that clients expect.
Unified Solver interface:
class Solver:
def solve(task_state: TaskState) -> SolverResult:
"""
Accepts a provider-agnostic task state containing:
- messages: list of Message(role, content)
- task_description: string
- current_state: any accumulated state
Returns a provider-agnostic result containing:
- output: string (the model's response)
- metadata: dict (token usage, model info, etc.)
"""
pass
Provider adapter pattern:
class ProviderSolver(Solver):
def solve(task_state: TaskState) -> SolverResult:
# Step 1: Convert evals format to provider format
provider_messages = self.convert_messages(task_state.messages)
# Step 2: Handle provider-specific constraints
provider_messages = self.enforce_constraints(provider_messages)
# Step 3: Call provider API with retry logic
response = self.api_call_with_retry(provider_messages)
# Step 4: Convert provider response back to evals format
return self.convert_response(response)
Message format translation examples:
Evals standard format:
{"role": "system", "content": "You are a helpful assistant."}
{"role": "user", "content": "What is 2+2?"}
Anthropic translation:
system = "You are a helpful assistant." # extracted to separate parameter
messages = [{"role": "user", "content": "What is 2+2?"}]
Gemini translation:
messages = [
{"role": "user", "parts": [{"text": "You are a helpful assistant.\n\nWhat is 2+2?"}]}
]
# system message merged into first user message; role "assistant" -> "model"
Retry logic with exponential backoff:
def api_call_with_retry(request, max_retries=5):
for attempt in range(max_retries):
try:
return provider_api.call(request)
except RateLimitError:
wait_time = base_delay * (2 ^ attempt) + random_jitter()
sleep(wait_time)
except SafetyFilterError as e:
return SolverResult(output="[blocked]", error=e)
raise MaxRetriesExceeded()
Provider capability matrix:
Feature | OpenAI | Anthropic | Gemini | Together
---------------------+--------+-----------+--------+---------
System role | Yes | Separate | Merge | Yes
Message alternation | No | Required | Required| No
Tool/function calls | Yes | Yes | Yes | Partial
Streaming | Yes | Yes | Yes | Yes
Image inputs | Yes | Yes | Yes | Partial
Safety filters | Mild | Moderate | Configurable | Varies
The adapter pattern ensures that none of these differences are visible to the evaluation harness, enabling write-once, evaluate-everywhere benchmark implementations.