Jump to content

Connect SuperML | Leeroopedia MCP: Equip your AI agents with best practices, code verification, and debugging knowledge. Powered by Leeroo — building Organizational Superintelligence. Contact us at founders@leeroo.com.

Implementation:LMCache LMCache VLLM Integration Utils

From Leeroopedia


Knowledge Sources
Domains vLLM Integration, Configuration Management
Last Updated 2026-02-09 00:00 GMT

Overview

This module provides utility functions for integrating LMCache with vLLM, including configuration loading, metadata creation, multimodal hash handling, and device detection.

Description

The vllm/utils.py module is a central utility file for the LMCache-vLLM integration layer. It implements thread-safe singleton configuration loading from files or environment variables (with optional remote configuration fetching), creates LMCache metadata objects from vLLM configuration, converts multimodal hash identifiers into 16-bit integers for token ID embedding, detects the appropriate torch device (CUDA or XPU), and calculates local rank/world size for multi-node tensor parallel deployments.

Usage

Import these utilities when building a vLLM KV connector or adapter that needs to initialize LMCache. The lmcache_get_or_create_config function is the primary entry point for configuration, and create_lmcache_metadata constructs the required metadata from vLLM's config objects.

Code Reference

Source Location

Signature

def is_false(value: str) -> bool: ...

def lmcache_get_or_create_config() -> LMCacheEngineConfig: ...

def hex_hash_to_int16(s: str) -> int: ...

def apply_mm_hashes_to_token_ids(
    token_ids: torch.Tensor,
    mm_hashes: list[str],
    mm_positions: list["PlaceholderRange"],
) -> torch.Tensor: ...

def mla_enabled(model_config: "ModelConfig") -> bool: ...

def create_lmcache_metadata(
    vllm_config=None, model_config=None, parallel_config=None,
    cache_config=None, role=None,
) -> tuple: ...

def extract_mm_features(
    request: "Request", modify: bool = False
) -> Tuple[list[str], list["PlaceholderRange"]]: ...

def get_size_bytes(
    shapes: list[torch.Size], kv_dtypes: list[torch.dtype]
) -> int: ...

def get_vllm_torch_dev() -> tuple: ...

def calculate_local_rank_and_world_size(
    vllm_config: "VllmConfig"
) -> Tuple[int, int]: ...

Import

from lmcache.integration.vllm.utils import (
    lmcache_get_or_create_config,
    create_lmcache_metadata,
    apply_mm_hashes_to_token_ids,
    hex_hash_to_int16,
    mla_enabled,
    extract_mm_features,
    get_size_bytes,
    get_vllm_torch_dev,
    calculate_local_rank_and_world_size,
)

I/O Contract

Inputs

Name Type Required Description
vllm_config VllmConfig No Top-level vLLM configuration object (alternative to individual configs)
model_config ModelConfig No vLLM model configuration (used if vllm_config is not provided)
parallel_config ParallelConfig No vLLM parallel configuration (used if vllm_config is not provided)
cache_config CacheConfig No vLLM cache configuration (used if vllm_config is not provided)
role str No Role identifier for the LMCache metadata
token_ids torch.Tensor Yes (for apply_mm_hashes_to_token_ids) Tensor of token IDs to modify with multimodal hashes
mm_hashes list[str] Yes (for apply_mm_hashes_to_token_ids) List of multimodal hash identifier strings
mm_positions list[PlaceholderRange] Yes (for apply_mm_hashes_to_token_ids) Corresponding placeholder positions for multimodal tokens

Outputs

Name Type Description
config LMCacheEngineConfig Singleton LMCache engine configuration loaded from file or environment
metadata, config tuple(LMCacheMetadata, LMCacheEngineConfig) Metadata and config derived from vLLM configuration
int16_hash int 16-bit integer hash derived from a multimodal identifier string
token_ids torch.Tensor Modified token IDs with multimodal placeholders overwritten
torch_dev, dev_name tuple Torch device module and device name string ("cuda" or "xpu")
local_worker_id, local_world_size Tuple[int, int] Local rank and local world size for the worker

Usage Examples

from lmcache.integration.vllm.utils import (
    lmcache_get_or_create_config,
    create_lmcache_metadata,
    hex_hash_to_int16,
)

# Load or create LMCache configuration (thread-safe singleton)
config = lmcache_get_or_create_config()
print(f"Chunk size: {config.chunk_size}")

# Create metadata from vLLM config
metadata, config = create_lmcache_metadata(vllm_config=vllm_config)

# Convert a multimodal hash to a 16-bit integer
hash_val = hex_hash_to_int16("chatcmpl-abc123-image-0")

Page Connections

Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment