Implementation:LMCache LMCache MP Cache Engine

Knowledge Sources	LMCache
Domains	KV Cache Management, GPU Computing, Multiprocess Architecture
Last Updated	2026-02-09 00:00 GMT

Overview

MPCacheEngine is the multiprocess cache engine that manages GPU-to-CPU KV cache transfers, storage, retrieval, and lookup operations, served over a ZMQ message queue.

Description

The MPCacheEngine class provides the core logic for a shared cache server in LMCache's multiprocess architecture. It maintains a registry of GPUCacheContext objects per GPU instance, where each context holds KV cache tensor pointers, pre-computed slot mappings, a temporary GPU transfer buffer, and dedicated CUDA/CuPy streams. The engine supports store (GPU-to-CPU transfer with async D2H memcpy), retrieve (CPU-to-GPU transfer with H2D memcpy), lookup (prefix-based key search with prefetch), and register_kv_cache/unregister_kv_cache for dynamic GPU context management. The run_cache_server function wires the engine to a MessageQueueServer with typed handlers for each request type. The GPUCacheContext helper class handles both MHA (5D) and MLA (3D) KV cache tensor layouts.

Usage

Use this module to run a standalone multiprocess KV cache server that can be shared across multiple vLLM worker processes. Workers communicate with the server via the MessageQueueClient, sending store/retrieve/lookup requests over ZMQ.

Code Reference

Source Location

• Repository: LMCache
• File: lmcache/v1/multiprocess/server.py
• Lines: 1-677

Signature

class GPUCacheContext:
    def __init__(self, kv_caches: KVCache, lmcache_chunk_size: int = 256) -> None: ...
    @property
    def dtype(self) -> torch.dtype: ...
    @property
    def device(self) -> torch.device: ...
    def get_tmp_gpu_buffer(self, num_tokens: int) -> torch.Tensor: ...
    def get_slot_mapping_tensor(self, gpu_block_ids: list[int]) -> torch.Tensor: ...
    def get_kv_buffer_shape(self, num_tokens: int) -> torch.Size: ...

class MPCacheEngine:
    def __init__(self, storage_manager_config: StorageManagerConfig,
                 chunk_size: int = 256) -> None: ...
    def register_kv_cache(self, instance_id: int, kv_caches: KVCache) -> None: ...
    def unregister_kv_cache(self, instance_id: int) -> None: ...
    def store(self, ipc_keys: list[IPCCacheEngineKey], instance_id: int,
              gpu_block_ids: list[int], event_ipc_handle: bytes) -> tuple[bytes, bool]: ...
    def retrieve(self, ipc_keys: list[IPCCacheEngineKey], instance_id: int,
                 gpu_block_ids: list[int], event_ipc_handle: bytes) -> tuple[bytes, list[bool]]: ...
    def lookup(self, ipc_keys: list[IPCCacheEngineKey],
               lock: bool | None = None) -> list[bool]: ...
    def clear(self) -> None: ...
    def close(self) -> None: ...

def run_cache_server(
    storage_manager_config: StorageManagerConfig,
    host: str = "localhost", port: int = 5555,
    chunk_size: int = 256, max_workers: int = 1,
    return_engine: bool = False,
) -> Optional[tuple[MessageQueueServer, MPCacheEngine]]: ...

Import

from lmcache.v1.multiprocess.server import MPCacheEngine, GPUCacheContext, run_cache_server

I/O Contract

Inputs

Name	Type	Required	Description
storage_manager_config	StorageManagerConfig	Yes	Configuration for the distributed storage manager (memory size, eviction policy)
chunk_size	int	No	Number of tokens per chunk for KV cache operations (default: 256)
instance_id	int	Yes (for store/retrieve)	GPU instance ID (e.g., process ID) identifying which GPU context to use
ipc_keys	list[IPCCacheEngineKey]	Yes	IPC-safe cache keys with model name, world size, worker ID, and chunk hash
gpu_block_ids	list[int]	Yes (for store/retrieve)	GPU block IDs specifying which vLLM KV cache blocks to transfer
event_ipc_handle	bytes	Yes (for store/retrieve)	IPC handle of a CUDA event to synchronize with the caller's stream
kv_caches	KVCache	Yes (for register)	List of CudaIPCWrapper objects wrapping vLLM GPU KV cache tensors

Outputs

Name	Type	Description
store()	tuple[bytes, bool]	CUDA event IPC handle and success flag
retrieve()	tuple[bytes, list[bool]]	CUDA event IPC handle and per-key success list
lookup()	list[bool]	Per-key presence flags (prefix-based, breaks at first miss)
get_chunk_size()	int	The configured chunk size

Usage Examples

# Run as a standalone ZMQ cache server
from lmcache.v1.multiprocess.server import run_cache_server
from lmcache.v1.distributed.config import StorageManagerConfig

config = StorageManagerConfig(...)
run_cache_server(
    storage_manager_config=config,
    host="localhost",
    port=5555,
    chunk_size=256,
    max_workers=4,
)
# Blocks until KeyboardInterrupt

# Or get server + engine for integration
server, engine = run_cache_server(
    storage_manager_config=config,
    host="localhost",
    port=5555,
    return_engine=True,
)