Implementation:Arize ai Phoenix Legacy LiteLLMModel

Overview

LiteLLMModel is a legacy model wrapper in the phoenix-evals package that provides an interface for using any LLM supported by the LiteLLM library within Phoenix LLM evaluations. It extends BaseModel and delegates to the litellm.completion() function, which provides a unified OpenAI-compatible interface to over 100 LLM providers. This wrapper does not support async execution and does not implement dynamic rate limiting, as the diverse range of supported providers makes reliable rate limit detection impractical.

LLM_Evaluation Model_Integration

Description

The LiteLLMModel class is implemented as a Python dataclass that extends the abstract BaseModel. Key characteristics include:

• Universal LLM interface: LiteLLM supports a wide range of providers including OpenAI, Anthropic, Cohere, Hugging Face, Ollama, Azure, AWS Bedrock, Google VertexAI, and many more through a single litellm.completion() call.
• Environment validation: At initialization, validates that required environment variables for the selected model are set using litellm.utils.validate_environment(), raising a RuntimeError with helpful guidance if keys are missing.
• No async support: The _async_generate_with_extra() method delegates to the synchronous _generate_with_extra(). Although LiteLLM provides an async interface, the wrapper avoids it because rate limit errors cannot be reliably caught and throttled across the diverse provider landscape.
• No rate limiting: Does not configure the RateLimiter with a specific error type, relying instead on LiteLLM's built-in retry mechanism via the num_retries parameter.
• Text-only prompts: Only supports text content parts; raises ValueError for image or audio content types.
• Deprecated field migration: The model_name field is deprecated in favor of model with automatic migration and deprecation warnings.
• Model-specific kwargs: Supports passing additional model-specific parameters through the model_kwargs dictionary.

Usage

# Configure the appropriate API key for your provider
# e.g., OPENAI_API_KEY, ANTHROPIC_API_KEY, OLLAMA_API_BASE, etc.

from phoenix.evals.models import LiteLLMModel

# Basic usage with an OpenAI model
model = LiteLLMModel(model="gpt-3.5-turbo")

# Use a local Ollama model
import os
os.environ["OLLAMA_API_BASE"] = "http://localhost:11434"
model = LiteLLMModel(model="ollama/llama3")

response = model("What is the meaning of life?")
print(response)

Code Reference

Source Location

Property	Value
Repository	Arize-ai/phoenix
File	packages/phoenix-evals/src/phoenix/evals/legacy/models/litellm.py
Lines	179
Module	phoenix.evals.legacy.models.litellm

Class Signature

@dataclass
class LiteLLMModel(BaseModel):
    model: str = "gpt-3.5-turbo"
    temperature: float = 0.0
    max_tokens: int = 1024
    top_p: float = 1
    num_retries: int = 0
    request_timeout: int = 60
    model_kwargs: Dict[str, Any] = field(default_factory=dict)
    # Deprecated
    model_name: Optional[str] = None

Constructor Parameters

Parameter	Type	Default	Description
model	str	"gpt-3.5-turbo"	The model name in LiteLLM format (e.g., "ollama/llama3", "anthropic/claude-3").
temperature	float	0.0	Sampling temperature.
max_tokens	int	1024	Maximum output tokens.
top_p	float	1	Nucleus sampling probability mass.
num_retries	int	0	Maximum retry attempts on rate limit, OpenAI, or service unavailable errors.
request_timeout	int	60	Maximum seconds to wait per request.
model_kwargs	Dict[str, Any]	{}	Additional model-specific parameters.
model_name	Optional[str]	None	Deprecated. Use model instead.

Key Methods

Method	Signature	Description
__post_init__	(self) -> None	Migrates deprecated fields and validates the LiteLLM environment.
_migrate_model_name	(self) -> None	Handles migration of model_name to model with deprecation warning.
_init_environment	(self) -> None	Imports LiteLLM and validates that required environment variables are set.
_generate_with_extra	(self, prompt, **kwargs) -> Tuple[str, ExtraInfo]	Synchronous generation via litellm.completion().
_async_generate_with_extra	async (self, prompt, **kwargs) -> Tuple[str, ExtraInfo]	Delegates to synchronous generation (no native async).
_extract_text	(self, response: ModelResponse) -> str	Extracts message content from LiteLLM's Choices response.
_extract_usage	(self, response: ModelResponse) -> Optional[Usage]	Extracts token usage from LiteLLM's Usage response object.
_parse_output	(self, response: ModelResponse) -> Tuple[str, ExtraInfo]	Combines text and usage extraction.
_get_messages_from_prompt	(self, prompt: MultimodalPrompt) -> List[Dict[str, str]]	Converts prompt to LiteLLM message format (text-only parts).

Import

from phoenix.evals.models import LiteLLMModel

I/O Contract

Direction	Type	Description
Input	Union[str, MultimodalPrompt]	A text string or multimodal prompt (only text parts supported).
Output	str	Generated text response.
Output (with extra)	Tuple[str, ExtraInfo]	Generated text paired with ExtraInfo containing optional Usage token counts.
Error	ImportError	Raised if litellm package is not installed.
Error	RuntimeError	Raised if required environment variables for the model are missing.
Error	ValueError	Raised if a non-text content type is provided in the prompt.

Usage Examples

Local Ollama Model

import os
from phoenix.evals.models import LiteLLMModel

os.environ["OLLAMA_API_BASE"] = "http://localhost:11434"

model = LiteLLMModel(
    model="ollama/llama3",
    temperature=0.0,
    max_tokens=512,
)
response = model("Explain microservices architecture.")
print(response)

With Retries

from phoenix.evals.models import LiteLLMModel

model = LiteLLMModel(
    model="gpt-3.5-turbo",
    num_retries=3,
    request_timeout=120,
)
response = model("What is the CAP theorem?")
print(response)

With Model-Specific Parameters

from phoenix.evals.models import LiteLLMModel

model = LiteLLMModel(
    model="anthropic/claude-3-haiku",
    model_kwargs={
        "stop": ["\n\n"],
        "presence_penalty": 0.5,
    },
)
response = model("List three machine learning algorithms.")
print(response)

Provider-Prefixed Models

from phoenix.evals.models import LiteLLMModel

# Using different providers via LiteLLM's naming convention
models = [
    LiteLLMModel(model="gpt-4"),                    # OpenAI
    LiteLLMModel(model="anthropic/claude-3-haiku"),  # Anthropic
    LiteLLMModel(model="ollama/mistral"),            # Local Ollama
    LiteLLMModel(model="bedrock/anthropic.claude-v2"),  # AWS Bedrock
]

Related Pages

• Arize_ai_Phoenix_Legacy_BaseModel - Abstract base class that LiteLLMModel extends
• Arize_ai_Phoenix_Legacy_OpenAIModel - Direct OpenAI wrapper (more features for OpenAI-specific use)
• Arize_ai_Phoenix_Legacy_AnthropicModel - Direct Anthropic wrapper (with rate limiting for Anthropic)
• Arize_ai_Phoenix_Legacy_BedrockModel - Direct AWS Bedrock wrapper (with rate limiting for Bedrock)
• Arize_ai_Phoenix_Legacy_VertexAIModel - Direct VertexAI wrapper