Implementation:LMCache LMCache Distributed API

Knowledge Sources	LMCache
Domains	Distributed Storage, Data Structures
Last Updated	2026-02-09 00:00 GMT

Overview

Defines the core data structures (ObjectKey and MemoryLayoutDesc) and conversion utilities used by the distributed storage manager.

Description

This module provides the fundamental data types for the LMCache distributed storage system. ObjectKey is a frozen dataclass that uniquely identifies a cached KV chunk via a content hash, model name, and a kv_rank bitmap encoding the parallelism configuration (world size, global rank, local world size, local rank). MemoryLayoutDesc describes the memory layout of stored objects as lists of tensor shapes and dtypes. The ipc_keys_to_object_keys function converts IPC-level cache engine keys into storage-level ObjectKeys, expanding keys with unspecified worker IDs into per-worker ObjectKeys for all workers in the world.

Usage

Use ObjectKey whenever referencing a specific KV cache chunk in the distributed storage layer. Use MemoryLayoutDesc to describe the expected tensor layout when allocating or reading memory. Use ipc_keys_to_object_keys to convert between the IPC communication layer's key representation and the storage layer's key representation.

Code Reference

Source Location

• Repository: LMCache
• File: lmcache/v1/distributed/api.py
• Lines: 1-183

Signature

@dataclass(frozen=True)
class ObjectKey:
    chunk_hash: bytes
    model_name: str
    kv_rank: int

    @staticmethod
    def IntHash2Bytes(chunk_hash: int) -> bytes: ...
    @staticmethod
    def Bytes2IntHash(chunk_hash: bytes) -> int: ...
    @staticmethod
    def ComputeKVRank(world_size: int, global_rank: int,
                      local_world_size: int, local_rank: int) -> int: ...

@dataclass(frozen=True)
class MemoryLayoutDesc:
    shapes: list[torch.Size]
    dtypes: list[torch.dtype]

def ipc_keys_to_object_keys(ipc_keys: list[IPCCacheEngineKey]) -> list[ObjectKey]: ...

Import

from lmcache.v1.distributed.api import ObjectKey, MemoryLayoutDesc, ipc_keys_to_object_keys

I/O Contract

Inputs

Name	Type	Required	Description
chunk_hash	bytes	Yes	Content hash identifying this particular KV cache chunk
model_name	str	Yes	Name of the model this chunk belongs to
kv_rank	int	Yes	Bitmap encoding the parallelism configuration for this chunk
world_size	int	Yes	Total number of workers (TP + PP) for ComputeKVRank
global_rank	int	Yes	Global worker ID (0 to world_size - 1) for ComputeKVRank
local_world_size	int	Yes	Local node world size (should not exceed 8) for ComputeKVRank
local_rank	int	Yes	Local node rank for ComputeKVRank
ipc_keys	list[IPCCacheEngineKey]	Yes	IPC keys to convert to ObjectKeys

Outputs

Name	Type	Description
ObjectKey	ObjectKey	Frozen dataclass uniquely identifying a chunk in distributed storage
MemoryLayoutDesc	MemoryLayoutDesc	Describes tensor shapes and dtypes for memory layout
storage_keys	list[ObjectKey]	Converted list of ObjectKeys from IPC keys

Usage Examples

from lmcache.v1.distributed.api import ObjectKey, MemoryLayoutDesc
import torch

# Create an object key with explicit kv_rank
kv_rank = ObjectKey.ComputeKVRank(
    world_size=8,
    global_rank=5,
    local_world_size=4,
    local_rank=1,
)
key = ObjectKey(
    chunk_hash=b"\x00\x01\x02\x03",
    model_name="llama-7b",
    kv_rank=kv_rank,
)

# Define a memory layout
layout = MemoryLayoutDesc(
    shapes=[torch.Size([2, 32, 128])],
    dtypes=[torch.float16],
)

# Convert IPC keys to storage keys
from lmcache.v1.distributed.api import ipc_keys_to_object_keys
storage_keys = ipc_keys_to_object_keys(ipc_key_list)