Implementation:BerriAI Litellm Batch Utils

Attribute	Value
Sources	litellm/router_utils/batch_utils.py
Domains	Router, Utilities, Batch Processing, File Handling
last_updated	2026-02-15 16:00 GMT

Overview

The Batch Utils module provides utilities for handling JSONL batch files in the LiteLLM Router, including model name replacement in batch request files and metadata variable name resolution.

Description

This module contains helper functions and classes used during batch processing through the router. The primary functionality includes: (1) InMemoryFile, a BytesIO subclass that carries a filename and content type for in-memory file operations; (2) parse_jsonl_with_embedded_newlines, a parser that correctly handles JSONL files where JSON string values contain literal newline characters; (3) replace_model_in_jsonl, which rewrites the body.model field in each line of a JSONL batch file to match the deployment model name (required by Azure Batch API); (4) should_replace_model_in_jsonl, which determines whether model replacement is needed based on the file purpose; and (5) _get_router_metadata_variable_name, which resolves whether metadata should use the "metadata" or "litellm_metadata" key based on the calling function.

Usage

Import these utilities when working with batch file uploads through the LiteLLM Router, particularly for Azure Batch API compatibility.

Code Reference

Source Location

litellm/router_utils/batch_utils.py

Classes

class InMemoryFile(io.BytesIO):
    def __init__(self, content: bytes, name: str, content_type: str = "application/jsonl"):

Functions

Function	Signature	Description
parse_jsonl_with_embedded_newlines	def parse_jsonl_with_embedded_newlines(content: str) -> List[dict]	Parses JSONL content handling embedded newlines in JSON string values
should_replace_model_in_jsonl	def should_replace_model_in_jsonl(purpose: OpenAIFilesPurpose) -> bool	Returns True if the model name should be replaced for the given file purpose (only "batch")
replace_model_in_jsonl	def replace_model_in_jsonl(file_content: FileTypes, new_model_name: str) -> FileTypes	Replaces body.model in each JSONL line with the deployment model name
_get_router_metadata_variable_name	def _get_router_metadata_variable_name(function_name: Optional[str]) -> str	Returns "litellm_metadata" for batch/file/generic endpoints, "metadata" otherwise

Import

from litellm.router_utils.batch_utils import (
    InMemoryFile,
    parse_jsonl_with_embedded_newlines,
    replace_model_in_jsonl,
    should_replace_model_in_jsonl,
    _get_router_metadata_variable_name,
)

I/O Contract

Inputs (replace_model_in_jsonl)

Parameter	Type	Description
file_content	FileTypes	The JSONL file content as bytes, string, file-like object, tuple, or PathLike
new_model_name	str	The deployment model name to substitute into each JSONL line

Outputs (replace_model_in_jsonl)

Return Type	Description
FileTypes	Modified file content as an InMemoryFile with updated model names, or the original content if parsing fails or content is a PathLike

Inputs (parse_jsonl_with_embedded_newlines)

Parameter	Type	Description
content	str	Raw JSONL content string that may contain embedded newlines in JSON values

Outputs (parse_jsonl_with_embedded_newlines)

Return Type	Description
List[dict]	List of parsed JSON objects from the JSONL content

Usage Examples

from litellm.router_utils.batch_utils import replace_model_in_jsonl, should_replace_model_in_jsonl

# Check if model replacement is needed
if should_replace_model_in_jsonl(purpose="batch"):
    modified_file = replace_model_in_jsonl(
        file_content=b'{"body": {"model": "gpt-4", "messages": [{"role": "user", "content": "Hello"}]}}\n',
        new_model_name="my-azure-gpt4-deployment",
    )

# Parse JSONL with embedded newlines
from litellm.router_utils.batch_utils import parse_jsonl_with_embedded_newlines

content = '{"id":1,"msg":"Line 1\\nLine 2"}\n{"id":2,"msg":"test"}'
objects = parse_jsonl_with_embedded_newlines(content)
# Returns: [{"id":1,"msg":"Line 1\nLine 2"}, {"id":2,"msg":"test"}]

# Resolve metadata variable name
from litellm.router_utils.batch_utils import _get_router_metadata_variable_name

name = _get_router_metadata_variable_name("batch")  # Returns "litellm_metadata"
name = _get_router_metadata_variable_name("completion")  # Returns "metadata"

Related Pages

• BerriAI_Litellm_Pattern_Match_Deployments - Deployment pattern matching for model routing
• BerriAI_Litellm_Tag_Based_Routing - Tag-based deployment filtering