Implementation:BerriAI Litellm Router Init

Knowledge Sources	Domains	Last Updated
litellm repository	LLM Load Balancing, API Gateway	2026-02-15

Overview

Concrete tool for initializing a load-balanced LLM router provided by LiteLLM, implemented as the Router.__init__ method.

Description

The Router.__init__ method constructs the entire routing infrastructure in a single call. It:

• Accepts a model_list of deployment dictionaries and builds O(1) lookup indexes by model name and model ID.
• Configures a DualCache (in-memory + optional Redis) for cooldown tracking, usage metrics, and response caching.
• Initializes a CooldownCache with configurable cooldown duration and allowed failure thresholds.
• Sets up the selected routing strategy (simple-shuffle, least-busy, usage-based, latency-based, or cost-based).
• Validates and stores fallback chains: model-specific, context-window, content-policy, and default/wildcard fallbacks.
• Optionally instantiates a RouterBudgetLimiting handler when provider or deployment budget configuration is provided.
• Registers success and failure callbacks on the LiteLLM callback system for deployment health tracking.
• Instantiates a Scheduler for priority-based request queuing.
• Creates an OpenAI-compatible chat.completions.create interface via litellm.Chat.

Usage

Import and instantiate the Router directly:

from litellm import Router

router = Router(model_list=model_list, routing_strategy="latency-based-routing")

Code Reference

Source Location: litellm/router.py, lines 213-656

Signature:

class Router:
    def __init__(
        self,
        model_list: Optional[Union[List[DeploymentTypedDict], List[Dict[str, Any]]]] = None,
        assistants_config: Optional[AssistantsTypedDict] = None,
        redis_url: Optional[str] = None,
        redis_host: Optional[str] = None,
        redis_port: Optional[int] = None,
        redis_password: Optional[str] = None,
        cache_responses: Optional[bool] = False,
        cache_kwargs: dict = {},
        caching_groups: Optional[List[tuple]] = None,
        client_ttl: int = 3600,
        polling_interval: Optional[float] = None,
        default_priority: Optional[int] = None,
        num_retries: Optional[int] = None,
        max_fallbacks: Optional[int] = None,
        timeout: Optional[float] = None,
        stream_timeout: Optional[float] = None,
        default_litellm_params: Optional[dict] = None,
        default_max_parallel_requests: Optional[int] = None,
        set_verbose: bool = False,
        debug_level: Literal["DEBUG", "INFO"] = "INFO",
        default_fallbacks: Optional[List[str]] = None,
        fallbacks: List = [],
        context_window_fallbacks: List = [],
        content_policy_fallbacks: List = [],
        model_group_alias: Optional[Dict[str, Union[str, RouterModelGroupAliasItem]]] = {},
        enable_pre_call_checks: bool = False,
        enable_tag_filtering: bool = False,
        retry_after: int = 0,
        retry_policy: Optional[Union[RetryPolicy, dict]] = None,
        model_group_retry_policy: Dict[str, RetryPolicy] = {},
        allowed_fails: Optional[int] = None,
        allowed_fails_policy: Optional[AllowedFailsPolicy] = None,
        cooldown_time: Optional[float] = None,
        disable_cooldowns: Optional[bool] = None,
        routing_strategy: Literal[
            "simple-shuffle", "least-busy", "usage-based-routing",
            "latency-based-routing", "cost-based-routing", "usage-based-routing-v2",
        ] = "simple-shuffle",
        routing_strategy_args: dict = {},
        provider_budget_config: Optional[GenericBudgetConfigType] = None,
        alerting_config: Optional[AlertingConfig] = None,
        router_general_settings: Optional[RouterGeneralSettings] = RouterGeneralSettings(),
        ignore_invalid_deployments: bool = False,
    ) -> None:

Import:

from litellm import Router
# or
from litellm.router import Router

I/O Contract

Key Inputs

Input Parameter	Type	Required	Description
model_list	Optional[List[Dict]]	No	List of deployment configurations; each dict has model_name and litellm_params
routing_strategy	Literal[...]	No	Algorithm for selecting deployments; defaults to "simple-shuffle"
num_retries	Optional[int]	No	Number of retries on failure; defaults to OpenAI default max retries
fallbacks	List	No	Model-specific fallback chains, e.g., [{"gpt-4": ["gpt-3.5-turbo"]}]
default_fallbacks	Optional[List[str]]	No	Wildcard fallbacks applied to all model groups
timeout	Optional[float]	No	Request timeout in seconds
cooldown_time	Optional[float]	No	Seconds to remove a failed deployment from the pool
retry_policy	Optional[RetryPolicy]	No	Per-exception-type retry counts
allowed_fails	Optional[int]	No	Failure count threshold before cooldown
provider_budget_config	Optional[dict]	No	Per-provider spend limits and time periods
redis_url	Optional[str]	No	Redis URL for distributed cache (enables cross-instance state)

Output

Output	Type	Description
Router instance	Router	Fully initialized router ready to handle completion(), acompletion(), and other LLM API calls

Usage Examples

Basic initialization with two Azure deployments and a fallback:

from litellm import Router

model_list = [
    {
        "model_name": "gpt-4",
        "litellm_params": {
            "model": "azure/gpt-4-east",
            "api_key": "sk-azure-east",
            "api_base": "https://east.openai.azure.com",
            "api_version": "2024-02-15",
        },
    },
    {
        "model_name": "gpt-4",
        "litellm_params": {
            "model": "azure/gpt-4-west",
            "api_key": "sk-azure-west",
            "api_base": "https://west.openai.azure.com",
            "api_version": "2024-02-15",
        },
    },
    {
        "model_name": "gpt-3.5-turbo",
        "litellm_params": {
            "model": "gpt-3.5-turbo",
            "api_key": "sk-openai-xxx",
        },
    },
]

router = Router(
    model_list=model_list,
    routing_strategy="latency-based-routing",
    num_retries=3,
    fallbacks=[{"gpt-4": ["gpt-3.5-turbo"]}],
    cooldown_time=30,
    timeout=60.0,
)

Initialization with retry policy and provider budgets:

from litellm import Router
from litellm.types.router import RetryPolicy

router = Router(
    model_list=model_list,
    retry_policy=RetryPolicy(
        RateLimitErrorRetries=5,
        TimeoutErrorRetries=3,
        InternalServerErrorRetries=2,
    ),
    provider_budget_config={
        "openai": {"budget_limit": 100.0, "time_period": "1d"},
        "azure": {"budget_limit": 200.0, "time_period": "1d"},
    },
    redis_url="redis://localhost:6379",
)

Related Pages

• Principle:BerriAI_Litellm_Router_Initialization