Implementation:LMCache LMCache Metadata

Knowledge Sources	LMCache
Domains	KV Cache, Model Configuration
Last Updated	2026-02-09 00:00 GMT

Overview

LMCacheMetadata is a dataclass that encapsulates all model and deployment metadata required by the LMCache system.

Description

This dataclass wraps the configuration attributes extracted from the northbound serving engine (such as vLLM), including model name, tensor parallelism settings, KV cache tensor shapes and data types, and layer group management. It provides utility methods for querying the KV cache shapes and dtypes across multiple layer groups, supporting both legacy single-group configurations and the newer KVLayerGroupsManager approach. The class also tracks distributed deployment information such as world size, worker IDs, and local variants for multi-node setups. MLA mode support, engine identity, and extra connector configuration are also represented.

Usage

Use this dataclass to pass model metadata throughout the LMCache system. It is typically constructed once during initialization from the serving engine's configuration and shared with cache engines, lookup clients, and storage backends.

Code Reference

Source Location

• Repository: LMCache
• File: lmcache/v1/metadata.py
• Lines: 1-114

Signature

@dataclass
class LMCacheMetadata:
    model_name: str
    world_size: int
    local_world_size: int
    worker_id: int
    local_worker_id: int
    kv_dtype: torch.dtype
    kv_shape: tuple[int, int, int, int, int]
    use_mla: bool = False
    role: Optional[str] = None
    served_model_name: Optional[str] = None
    chunk_size: int = 256
    kv_layer_groups_manager: KVLayerGroupsManager = field(default_factory=KVLayerGroupsManager)
    engine_id: Optional[str] = None
    kv_connector_extra_config: Optional[dict] = None

    def is_first_rank(self) -> bool: ...
    def get_dtypes(self) -> list[torch.dtype]: ...
    def get_shapes(self, num_tokens: Optional[int] = None) -> list[torch.Size]: ...
    def get_num_groups(self) -> int: ...

Import

from lmcache.v1.metadata import LMCacheMetadata

I/O Contract

Inputs

Name	Type	Required	Description
model_name	str	Yes	Name of the LLM model
world_size	int	Yes	Total number of distributed workers
local_world_size	int	Yes	Number of workers on the local host
worker_id	int	Yes	Global worker identifier
local_worker_id	int	Yes	Local worker identifier on the current host
kv_dtype	torch.dtype	Yes	Data type of KV cache tensors (deprecated, prefer kv_layer_groups_manager)
kv_shape	tuple[int, int, int, int, int]	Yes	Shape of KV tensors as (num_layer, 2, chunk_size, num_kv_head, head_size) (deprecated)
use_mla	bool	No	Whether to use Multi-head Latent Attention (default False)
chunk_size	int	No	Number of tokens per chunk (default 256)
num_tokens	Optional[int]	No	Override token count when calling get_shapes (defaults to chunk_size)

Outputs

Name	Type	Description
is_first_rank	bool	True if the current worker is rank 0
get_dtypes	list[torch.dtype]	List of KV cache dtypes per layer group
get_shapes	list[torch.Size]	List of KV cache shapes per layer group, each as [kv_size, num_layers, num_tokens, hidden_dim_size]
get_num_groups	int	Number of KV layer groups (1 for legacy configurations)

Usage Examples

from lmcache.v1.metadata import LMCacheMetadata
import torch

metadata = LMCacheMetadata(
    model_name="llama-7b",
    world_size=4,
    local_world_size=4,
    worker_id=0,
    local_worker_id=0,
    kv_dtype=torch.float16,
    kv_shape=(32, 2, 256, 32, 128),
)

# Check rank
if metadata.is_first_rank():
    print("This is the first rank")

# Get shapes for a specific token count
shapes = metadata.get_shapes(num_tokens=512)