Implementation:BerriAI Litellm Batch Completion

Attribute	Value
Sources	litellm/batch_completion/main.py
Domains	Batch Processing, Completion, Parallelism, Multi-Model
Last Updated	2026-02-15 16:00 GMT

Overview

The batch_completion module provides functions for sending multiple LLM completion requests in parallel, supporting both batching across message lists and racing requests across multiple models.

Description

This module contains three main functions for batch processing LLM requests:

• batch_completion - Takes a list of message lists and sends them in parallel using a ThreadPoolExecutor. For VLLM providers, it delegates to the specialized vllm_handler.batch_completions method. For all other providers, it chunks the messages into sub-batches of 100 and dispatches them as individual litellm.completion calls. Exceptions from individual calls are captured and included in the results list rather than raising.

• batch_completion_models - Sends the same request to multiple models concurrently and returns the first successful response. Supports two modes: a models list (simple model names) or a deployments list (full deployment configurations with model, API base, etc.). Failed models are skipped, and the function falls back to the next available response.

• batch_completion_models_all_responses - Similar to batch_completion_models, but collects and returns responses from all models that respond successfully, rather than just the first one.

Usage

Import these functions when you need to process multiple completion requests in parallel, race requests across multiple models for the fastest response, or collect responses from multiple models for comparison.

Code Reference

Source Location

litellm/batch_completion/main.py

Signature

def batch_completion(
    model: str, messages: List = [], functions: Optional[List] = None,
    function_call: Optional[str] = None, temperature: Optional[float] = None,
    top_p: Optional[float] = None, n: Optional[int] = None,
    stream: Optional[bool] = None, stop=None, max_tokens: Optional[int] = None,
    presence_penalty: Optional[float] = None, frequency_penalty: Optional[float] = None,
    logit_bias: Optional[dict] = None, user: Optional[str] = None,
    deployment_id=None, request_timeout: Optional[int] = None,
    timeout: Optional[int] = 600, max_workers: Optional[int] = 100,
    **kwargs,
) -> list

def batch_completion_models(*args, **kwargs) -> Optional[Any]

def batch_completion_models_all_responses(*args, **kwargs) -> list

Import

from litellm.batch_completion.main import (
    batch_completion,
    batch_completion_models,
    batch_completion_models_all_responses,
)

I/O Contract

Inputs

Parameter	Type	Description
model	str	The model identifier (may include provider prefix, e.g., "vllm/my-model").
messages	List[List]	A list of message lists, each representing one completion request.
max_workers	Optional[int]	Maximum number of parallel threads. Defaults to 100.
timeout	Optional[int]	Overall timeout in seconds. Defaults to 600.
models (kwarg)	List[str]	List of model names for multi-model functions.
deployments (kwarg)	List[dict]	List of deployment configurations for multi-model functions.
Standard OpenAI params	various	temperature, top_p, max_tokens, stop, frequency_penalty, presence_penalty, etc.

Outputs

Function	Return Type	Description
batch_completion	list	A list of completion results (or exceptions for failed requests).
batch_completion_models	Optional[Any]	The first successful response from any model, or None.
batch_completion_models_all_responses	list	All successful responses from all models.

Usage Examples

import litellm
from litellm.batch_completion.main import batch_completion, batch_completion_models

# Batch multiple message lists to the same model
results = batch_completion(
    model="gpt-3.5-turbo",
    messages=[
        [{"role": "user", "content": "What is 2+2?"}],
        [{"role": "user", "content": "What is the capital of France?"}],
        [{"role": "user", "content": "Tell me a joke."}],
    ],
    max_workers=10,
)

for result in results:
    if isinstance(result, Exception):
        print(f"Error: {result}")
    else:
        print(result.choices[0].message.content)

# Race multiple models and return the first response
response = batch_completion_models(
    models=["gpt-3.5-turbo", "claude-3-haiku-20240307"],
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

Related Pages

• BerriAI_Litellm_Budget_Manager - Budget tracking that can be used alongside batch completion
• BerriAI_Litellm_Timeout_Decorator - Timeout decorator for individual completion calls