Implementation:LMCache LMCache HTTP Server

Knowledge Sources	LMCache
Domains	HTTP API, KV Cache Management, Server Infrastructure
Last Updated	2026-02-09 00:00 GMT

Overview

The HTTP server module provides a FastAPI-based REST API for managing KV cache tensors, integrating with the multiprocess ZMQ cache engine for storage and retrieval operations.

Description

This module implements an HTTP server using FastAPI and Uvicorn that exposes REST endpoints for listing, retrieving, downloading, and uploading KV cache tensors. On startup it initializes a ZMQ-backed MPCacheEngine via the run_cache_server function, managed through FastAPI's lifespan context. The server supports multiple tensor serialization formats (numpy .npy, JSON with base64 encoding, and safetensors), configurable hash encodings (hex and base64url), and provides metadata inspection endpoints. A ServerConfig dataclass holds configuration for both the HTTP server and the underlying ZMQ backend. The module includes helper functions for tensor serialization, hash encoding/decoding, and key-based search across the storage engine.

Usage

Use this module to expose an HTTP API for debugging, inspecting, or remotely managing KV cache data in an LMCache deployment. It can be run standalone via command-line arguments or integrated as part of a larger service.

Code Reference

Source Location

• Repository: LMCache
• File: lmcache/v1/multiprocess/http_server.py
• Lines: 1-807

Signature

@dataclass
class ServerConfig:
    zmq_host: str = "localhost"
    zmq_port: int = 5555
    chunk_size: int = 256
    cpu_buffer_size: float = 5.0
    max_workers: int = 1
    def to_storage_manager_config(self) -> StorageManagerConfig: ...

class DownloadRequest(BaseModel):
    chunk_hash: str
    model_name: Optional[str] = None
    world_size: Optional[int] = None
    worker_id: Optional[int] = None
    hash_encoding: HashEncoding = "hex"

def run_http_server(
    host: str = "0.0.0.0", port: int = 8000,
    zmq_host: str = "localhost", zmq_port: int = 5555,
    chunk_size: int = 256, cpu_buffer_size: float = 5.0,
    max_workers: int = 1,
) -> None: ...

# FastAPI endpoints
async def root(request: Request) -> dict: ...
async def get_all_hashes(request: Request, encoding: HashEncoding = "hex") -> list: ...
async def get_kv_cache(request: Request, hash_str: str, ...) -> Response: ...
async def get_kv_cache_metadata(request: Request, hash_str: str, ...) -> JSONResponse: ...
async def download_kv_cache(request: Request, download_request: DownloadRequest) -> Response: ...
async def set_kv_cache(request: Request, chunk_hash: str, safetensors: UploadFile, ...) -> JSONResponse: ...

Import

from lmcache.v1.multiprocess.http_server import run_http_server, app, ServerConfig

I/O Contract

Inputs

Name	Type	Required	Description
host	str	No	HTTP server bind address (default: "0.0.0.0")
port	int	No	HTTP server port (default: 8000)
zmq_host	str	No	ZMQ backend host (default: "localhost")
zmq_port	int	No	ZMQ backend port (default: 5555)
chunk_size	int	No	Chunk size for KV cache operations (default: 256)
cpu_buffer_size	float	No	CPU buffer size in GB (default: 5.0)
hash_str	str	Yes (for GET endpoints)	Hash string identifying a KV cache entry
hash_encoding	HashEncoding	No	Hash encoding format: "hex" or "b64url" (default: "hex")
safetensors	UploadFile	Yes (for set endpoint)	Uploaded safetensors file containing tensor data

Outputs

Name	Type	Description
GET /	dict	Status response with service name
GET /api/v1/all_hashes	list[str]	List of all chunk hashes in the configured encoding
GET /api/v1/kv_cache/{hash_str}	Response	Raw .npy bytes or JSON with base64-encoded tensor data
GET /api/v1/kv_cache/{hash_str}/metadata	JSONResponse	Tensor metadata including shape, dtype, and size_bytes
POST /api/v1/kv_cache/download	Response	Safetensors file download of the requested KV cache tensor
POST /api/v1/kv_cache/set	JSONResponse	Success status with mode (overwrite), shape, and dtype

Usage Examples

# Run the HTTP server from command line
# python -m lmcache.v1.multiprocess.http_server --host 0.0.0.0 --port 8000 --zmq-port 5555

# Or programmatically
from lmcache.v1.multiprocess.http_server import run_http_server

run_http_server(
    host="0.0.0.0",
    port=8000,
    zmq_host="localhost",
    zmq_port=5555,
    chunk_size=256,
    cpu_buffer_size=5.0,
)

# Client-side usage with requests
import requests
response = requests.get("http://localhost:8000/api/v1/all_hashes?encoding=hex")
hashes = response.json()

response = requests.get(f"http://localhost:8000/api/v1/kv_cache/{hashes[0]}")