Implementation:BerriAI Litellm Scheduler

Attribute	Value
Sources	litellm/scheduler.py
Domains	Router, Scheduling, Priority Queue, Flow Control
last_updated	2026-02-15 16:00 GMT

Overview

The Scheduler module implements a priority-based request queue that controls the order in which LLM requests are processed, supporting priorities from 0 (highest) to 255 (lowest).

Description

This module provides the Scheduler class, which manages a per-model-group priority queue backed by a DualCache (supporting both in-memory and optional Redis storage for cross-instance coordination). Requests are represented as FlowItem objects with a priority, request ID, and model name. The scheduler uses Python's heapq module to maintain heap ordering, where lower priority values indicate higher priority. The poll method determines whether a request can proceed: if healthy deployments are available, all requests proceed; if no deployments are available, only the highest-priority request (top of queue) can proceed while others wait. The peek method checks queue position without modifying state, and remove_request handles cleanup for timed-out requests.

Usage

Import and use the Scheduler when implementing priority-based request handling in the LiteLLM Router. It is used internally by the router to manage request ordering under rate limit pressure.

Code Reference

Source Location

litellm/scheduler.py

Classes

class SchedulerCacheKeys(enum.Enum):
    queue = "scheduler:queue"
    default_in_memory_ttl = DEFAULT_IN_MEMORY_TTL

class FlowItem(BaseModel):
    priority: int  # Priority between 0 and 255
    request_id: str
    model_name: str

class Scheduler:
    cache: DualCache

    def __init__(
        self,
        polling_interval: Optional[float] = None,
        redis_cache: Optional[RedisCache] = None,
    ):

Key Methods

Method	Signature	Description
add_request	async def add_request(self, request: FlowItem)	Adds a request to the priority queue for its model group
poll	async def poll(self, id: str, model_name: str, health_deployments: list) -> bool	Returns True if the request can proceed (deployments available or at top of queue); pops from queue if at top with no deployments
peek	async def peek(self, id: str, model_name: str, health_deployments: list) -> bool	Returns True if the request is at the top of the queue without modifying state
remove_request	async def remove_request(self, request_id: str, model_name: str) -> None	Removes a specific request from the queue (e.g., on timeout)
get_queue	async def get_queue(self, model_name: str) -> list	Retrieves the current priority queue for a model group from cache
save_queue	async def save_queue(self, queue: list, model_name: str) -> None	Persists the updated queue back to cache
get_queue_status	def get_queue_status(self)	Returns the local in-memory queue state

Import

from litellm.scheduler import Scheduler, FlowItem, SchedulerCacheKeys

I/O Contract

Inputs (add_request)

Parameter	Type	Description
request	FlowItem	A Pydantic model with priority (0-255), request_id, and model_name

Inputs (poll)

Parameter	Type	Description
id	str	The request ID to check
model_name	str	The model group name
health_deployments	list	Currently healthy deployments for this model

Outputs (poll)

Return Type	Description
bool	True if request can proceed, False if it must wait

Usage Examples

from litellm.scheduler import Scheduler, FlowItem

scheduler = Scheduler(polling_interval=0.003)  # 3ms polling

# Add requests with different priorities
await scheduler.add_request(FlowItem(priority=0, request_id="req-1", model_name="gpt-4"))
await scheduler.add_request(FlowItem(priority=128, request_id="req-2", model_name="gpt-4"))

# Check if a request can proceed
can_proceed = await scheduler.poll(
    id="req-1",
    model_name="gpt-4",
    health_deployments=[{"model_info": {"id": "deploy-1"}}],
)
# Returns True (deployments available)

# When no deployments available, only top-priority request proceeds
can_proceed = await scheduler.poll(
    id="req-1",
    model_name="gpt-4",
    health_deployments=[],
)
# Returns True (req-1 is highest priority)

Related Pages

• BerriAI_Litellm_Lowest_TPM_RPM_V2_Strategy - TPM/RPM strategy that interacts with the scheduler
• BerriAI_Litellm_Least_Busy_Strategy - Load balancing strategy used alongside scheduling