Implementation:BerriAI Litellm Prompt Caching Cache

Attribute	Value
Sources	litellm/router_utils/prompt_caching_cache.py
Domains	Router, Utilities, Prompt Caching, Deployment Affinity
last_updated	2026-02-15 16:00 GMT

Overview

The Prompt Caching Cache module stores a mapping from prompt cache keys to deployment model IDs, enabling the router to route subsequent requests with the same cacheable prefix to the same deployment for optimal prompt cache hit rates.

Description

This module provides the PromptCachingCache class, which wraps the router's DualCache to maintain deployment affinity for prompt caching. It works by extracting the "cacheable prefix" from messages -- everything up to and including the last content block with a cache_control directive of type "ephemeral". This prefix is serialized and hashed (SHA-256) to produce a stable cache key. When a request completes successfully, the deployment's model ID is stored against this key (with a 5-minute TTL). On subsequent requests with the same cacheable prefix, the cache returns the previously used model ID so the router can preferentially route to it, maximizing provider-side prompt cache hits. The module handles both message-level and content-block-level cache_control directives.

Usage

Import PromptCachingCache when setting up the LiteLLM Router with prompt caching support. The router uses it internally to track which deployment handled which cacheable prefix.

Code Reference

Source Location

litellm/router_utils/prompt_caching_cache.py

Classes

class PromptCachingCacheValue(TypedDict):
    model_id: str

class PromptCachingCache:
    def __init__(self, cache: DualCache):

Key Methods

Method	Signature	Description
`extract_cacheable_prefix`	`@staticmethod def extract_cacheable_prefix(messages: List[AllMessageValues]) -> List[AllMessageValues]`	Extracts messages up to and including the last `cache_control` block
`get_prompt_caching_cache_key`	`@staticmethod def get_prompt_caching_cache_key(messages, tools) -> Optional[str]`	Generates a SHA-256-based cache key from the cacheable prefix
`add_model_id`	`def add_model_id(self, model_id: str, messages, tools) -> None`	Stores a model ID against the cacheable prefix (sync, 5-min TTL)
`async_add_model_id`	`async def async_add_model_id(self, model_id: str, messages, tools) -> None`	Stores a model ID against the cacheable prefix (async, 5-min TTL)
`get_model_id`	`def get_model_id(self, messages, tools) -> Optional[PromptCachingCacheValue]`	Retrieves the cached model ID for the cacheable prefix (sync)
`async_get_model_id`	`async def async_get_model_id(self, messages, tools) -> Optional[PromptCachingCacheValue]`	Retrieves the cached model ID for the cacheable prefix (async)
`serialize_object`	`@staticmethod def serialize_object(obj: Any) -> Any`	Serializes Pydantic models, dicts, and lists for stable hashing

Import

from litellm.router_utils.prompt_caching_cache import PromptCachingCache, PromptCachingCacheValue

I/O Contract

Inputs (async_get_model_id)

Parameter	Type	Description
`messages`	`Optional[List[AllMessageValues]]`	Chat messages potentially containing `cache_control` blocks
`tools`	`Optional[List[ChatCompletionToolParam]]`	Tool definitions to include in the cache key

Outputs (async_get_model_id)

Return Type	Description
`Optional[PromptCachingCacheValue]`	A `TypedDict` with `model_id: str` if found, or `None`

Usage Examples

from litellm.caching.caching import DualCache
from litellm.router_utils.prompt_caching_cache import PromptCachingCache

cache = DualCache()
prompt_cache = PromptCachingCache(cache=cache)

messages = [
    {"role": "system", "content": [
        {"type": "text", "text": "You are a helpful assistant.", "cache_control": {"type": "ephemeral"}},
    ]},
    {"role": "user", "content": "What is 2+2?"},
]

# After a successful call to deployment "deploy-abc"
prompt_cache.add_model_id(
    model_id="deploy-abc",
    messages=messages,
    tools=None,
)

# On subsequent request with same system prompt
cached = prompt_cache.get_model_id(messages=messages, tools=None)
if cached:
    preferred_deployment_id = cached["model_id"]  # "deploy-abc"

Related Pages

BerriAI_Litellm_Lowest_TPM_RPM_V2_Strategy - Deployment selection that can incorporate prompt cache affinity
BerriAI_Litellm_Pattern_Match_Deployments - Pattern-based deployment routing

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment