Principle:BerriAI Litellm Cache Management

Knowledge Sources	Cache lifecycle management; operational monitoring patterns; HTTP API design for infrastructure services
Domains	Caching, Operations, API Design, LLM Infrastructure
Last Updated	2026-02-15

Overview

Cache management is the set of operational capabilities for monitoring cache health, selectively deleting cached entries, inspecting backend state, and performing bulk cache invalidation through administrative HTTP endpoints.

Description

A production caching system requires more than just reading and writing entries. Cache management encompasses the day-two operational concerns that arise once a cache is deployed:

• Health monitoring (ping): Before relying on cached responses, operators need to verify that the cache backend is reachable and functioning. A health check endpoint tests connectivity (e.g., a Redis PING), performs a test write, and reports the cache type, configuration parameters (with sensitive values masked), and overall health status.
• Selective key deletion: When a cached response becomes stale or incorrect (for example, after a model update or a prompt template change), operators need to delete specific cache entries by key without affecting the rest of the cache. The x-litellm-cache-key response header exposes the cache key used for each request, making targeted deletion straightforward.
• Backend introspection: For Redis-based caches, operators may need to inspect server-level metrics such as the number of connected clients, memory usage, and server version. This helps diagnose performance issues and plan capacity.
• Bulk invalidation (flush): In extreme cases (e.g., a complete model migration, a security incident requiring all cached responses to be discarded), a flush-all operation clears every entry in the cache. This is a destructive operation that requires authentication and should be used with caution.
• Authentication and authorization: All management endpoints must be protected behind the same authentication layer as the proxy's main API, ensuring that only authorized administrators can inspect or modify the cache.
• Sensitive data masking: Cache configuration parameters (host, port, password, Redis kwargs) may contain secrets. Health check responses must mask these values before returning them to the caller.

Usage

Use cache management capabilities when:

• You are deploying a LiteLLM proxy in production and need to verify that the cache backend is correctly configured and reachable.
• You need to invalidate specific cached responses without restarting the proxy or flushing the entire cache.
• You are debugging cache performance and need to inspect Redis server metrics and client connections.
• You need to perform an emergency cache flush after a model or data change.
• You are building a monitoring dashboard or alerting system that polls the cache health endpoint.

Theoretical Basis

Cache management follows the control plane / data plane separation pattern common in infrastructure systems. The data plane handles the high-throughput read/write operations (cache get/set), while the control plane provides low-frequency administrative operations (health checks, deletions, flushes). Exposing the control plane as HTTP endpoints follows RESTful conventions and integrates naturally with orchestration systems, load balancers, and health check mechanisms.

Pseudocode:

ROUTER /cache:
    -- All endpoints require authentication via API key

    ENDPOINT GET /ping:
        IF cache is not initialized:
            RETURN 503 "Cache not initialized"

        litellm_params = mask_sensitive(vars(cache))
        cleaned_params = extract_and_mask_backend_params(cache.backend)

        IF cache.type == REDIS:
            ping_result = AWAIT cache.ping()
            AWAIT cache.set("test_key", test_data)   -- verify write works
            RETURN {
                status: "healthy",
                cache_type: cache.type,
                ping_response: True,
                set_cache_response: "success",
                litellm_cache_params: litellm_params,
                health_check_cache_params: cleaned_params,
            }
        ELSE:
            RETURN {
                status: "healthy",
                cache_type: cache.type,
                litellm_cache_params: litellm_params,
            }

    ENDPOINT POST /delete:
        IF cache is not initialized:
            RETURN 503 "Cache not initialized"
        keys = request.body["keys"]     -- list of cache keys to delete
        IF cache.type == REDIS:
            AWAIT cache.delete_keys(keys)
            RETURN { status: "success" }
        ELSE:
            RETURN 500 "Only Redis supports key deletion"

    ENDPOINT GET /redis/info:
        IF cache is not initialized OR cache.type != REDIS:
            RETURN 503 / 500
        client_list = TRY cache.backend.client_list()
                      CATCH -> ["CLIENT LIST not available"]
        redis_info = cache.backend.info()
        RETURN {
            num_clients: len(client_list),
            clients: client_list,
            info: redis_info,
        }

    ENDPOINT POST /flushall:
        IF cache is not initialized:
            RETURN 503 "Cache not initialized"
        IF cache.type == REDIS:
            cache.backend.flushall()
            RETURN { status: "success" }
        ELSE:
            RETURN 500 "Only Redis supports flushing"

The key design properties are:

• Graceful degradation: The /redis/info endpoint handles the case where the CLIENT LIST command is unavailable (common on managed Redis services) by returning a descriptive message instead of failing.
• Idempotent reads: The /ping and /redis/info endpoints are safe to call repeatedly for monitoring without side effects (the test write in /ping uses a fixed test key).
• Destructive operations are POST: Both /delete and /flushall use the POST method, signalling that they have side effects and preventing accidental invocation via browser navigation or cached GET requests.
• Defense in depth: Authentication is enforced at the router level via dependency injection, ensuring every endpoint is protected regardless of how the router is mounted.

Related Pages

• Implementation:BerriAI_Litellm_Cache_Endpoints