Implementation:LMCache LMCache Standalone Starter

Knowledge Sources	LMCache
Domains	Standalone Deployment, Configuration Management, Cache Engine
Last Updated	2026-02-09 00:00 GMT

Overview

The standalone starter module provides a command-line entry point for running an LMCacheEngine instance without vLLM or GPU, supporting YAML/environment configuration, multi-group KV shape specifications, and optional API server integration.

Description

This module implements LMCacheStandaloneStarter, which initializes a full LMCacheEngine via a StandaloneLMCacheManager using mock broadcast functions and a mock GPU connector. It supports multiple layer groups with heterogeneous shapes and dtypes through the LayerGroupSpec dataclass and parse_kvcache_shape_spec parser, which accepts semicolon-delimited group specifications in the format (shape):dtype:layer_count. The starter loads configuration from YAML files or environment variables, applies command-line overrides, generates fixed-pattern KV cache tensors for testing, and runs the engine with signal handling and async event loop support. Helper functions include load_config, override_config_from_dict, parse_kv_shape, and composite shape calculators for multi-group setups.

Usage

Use this module to run LMCache as a standalone service for testing, debugging, or deployment scenarios where vLLM is not present. It is the main entry point when running python -m lmcache.v1.standalone.

Code Reference

Source Location

• Repository: LMCache
• File: lmcache/v1/standalone/__main__.py
• Lines: 1-585

Signature

class LayerGroupSpec:
    def __init__(self, layer_count: int, shape: Tuple[int, ...],
                 dtype: torch.dtype) -> None: ...

class LMCacheStandaloneStarter:
    def __init__(self, config: LMCacheEngineConfig, metadata: LMCacheMetadata,
                 layer_groups: List[LayerGroupSpec], device: str = "cpu") -> None: ...
    @property
    def lmcache_engine(self) -> LMCacheEngine: ...
    @property
    def api_server(self): ...
    def start(self) -> LMCacheEngine: ...
    def stop(self) -> None: ...
    async def run_forever(self) -> None: ...

def parse_kvcache_shape_spec(spec_str: str) -> List[LayerGroupSpec]: ...
def calculate_composite_kv_cache_shape(
    layer_groups: List[LayerGroupSpec]) -> Tuple[int, int, int, int, int]: ...
def get_composite_kv_dtype(layer_groups: List[LayerGroupSpec]) -> torch.dtype: ...
def load_config(config_file: Optional[str] = None) -> LMCacheEngineConfig: ...
def override_config_from_dict(config: LMCacheEngineConfig,
                              overrides: Dict[str, Any]) -> None: ...
def parse_kv_shape(shape_str: str) -> Tuple[int, int, int, int, int]: ...
def main() -> None: ...

Import

from lmcache.v1.standalone.__main__ import (
    LMCacheStandaloneStarter,
    LayerGroupSpec,
    parse_kvcache_shape_spec,
    load_config,
)

I/O Contract

Inputs

Name	Type	Required	Description
config	LMCacheEngineConfig	Yes	Engine configuration loaded from YAML file or environment variables
metadata	LMCacheMetadata	Yes	Metadata with model_name, world_size, worker_id, kv_shape, kv_dtype, and use_mla flag
layer_groups	List[LayerGroupSpec]	Yes	List of layer group specifications defining per-group shape, dtype, and layer count
device	str	No	Device for tensor allocation (default: "cpu")
--config	str (CLI)	No	Path to YAML configuration file
--model-name	str (CLI)	No	Model name for cache identification (default: "standalone_model")
--kvcache-shape-spec	str (CLI)	No	Multi-group KV shape specification (default: "(2,2,256,4,16):float16:2")
--kv-shape	str (CLI)	No	KV cache shape as comma-separated integers (default: "2,2,256,4,16")
--kv-dtype	str (CLI)	No	KV cache data type (default: "float16")
--use-mla	flag (CLI)	No	Enable MLA (Multi-Level Attention) mode

Outputs

Name	Type	Description
start()	LMCacheEngine	The initialized and running LMCacheEngine instance
lmcache_engine	LMCacheEngine	Property accessor for the underlying engine
parse_kvcache_shape_spec()	List[LayerGroupSpec]	Parsed list of layer group specifications from the spec string
calculate_composite_kv_cache_shape()	Tuple[int, int, int, int, int]	Composite (num_layers, kv_dim, num_blocks, max_num_heads, max_head_size)

Usage Examples

# Command-line usage
# python -m lmcache.v1.standalone --config config.yaml \
#     --model-name my_model --kvcache-shape-spec "(32,2,256,32,128):bfloat16:32"

# Programmatic usage
from lmcache.v1.standalone.__main__ import (
    LMCacheStandaloneStarter,
    parse_kvcache_shape_spec,
)
from lmcache.v1.config import LMCacheEngineConfig
from lmcache.v1.metadata import LMCacheMetadata

config = LMCacheEngineConfig.from_file("config.yaml")
layer_groups = parse_kvcache_shape_spec("(2,2,256,4,16):float16:2")

metadata = LMCacheMetadata(
    model_name="standalone_model",
    world_size=1,
    worker_id=0,
    kv_dtype=torch.float16,
    kv_shape=(2, 2, 256, 4, 16),
)

starter = LMCacheStandaloneStarter(config, metadata, layer_groups)
engine = starter.start()
# engine is now running and can be used for cache operations
starter.stop()