Implementation:LMCache LMCache VLLM Integration Utils
| Knowledge Sources | |
|---|---|
| Domains | vLLM Integration, Configuration Management |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
This module provides utility functions for integrating LMCache with vLLM, including configuration loading, metadata creation, multimodal hash handling, and device detection.
Description
The vllm/utils.py module is a central utility file for the LMCache-vLLM integration layer. It implements thread-safe singleton configuration loading from files or environment variables (with optional remote configuration fetching), creates LMCache metadata objects from vLLM configuration, converts multimodal hash identifiers into 16-bit integers for token ID embedding, detects the appropriate torch device (CUDA or XPU), and calculates local rank/world size for multi-node tensor parallel deployments.
Usage
Import these utilities when building a vLLM KV connector or adapter that needs to initialize LMCache. The lmcache_get_or_create_config function is the primary entry point for configuration, and create_lmcache_metadata constructs the required metadata from vLLM's config objects.
Code Reference
Source Location
- Repository: LMCache
- File: lmcache/integration/vllm/utils.py
- Lines: 1-343
Signature
def is_false(value: str) -> bool: ...
def lmcache_get_or_create_config() -> LMCacheEngineConfig: ...
def hex_hash_to_int16(s: str) -> int: ...
def apply_mm_hashes_to_token_ids(
token_ids: torch.Tensor,
mm_hashes: list[str],
mm_positions: list["PlaceholderRange"],
) -> torch.Tensor: ...
def mla_enabled(model_config: "ModelConfig") -> bool: ...
def create_lmcache_metadata(
vllm_config=None, model_config=None, parallel_config=None,
cache_config=None, role=None,
) -> tuple: ...
def extract_mm_features(
request: "Request", modify: bool = False
) -> Tuple[list[str], list["PlaceholderRange"]]: ...
def get_size_bytes(
shapes: list[torch.Size], kv_dtypes: list[torch.dtype]
) -> int: ...
def get_vllm_torch_dev() -> tuple: ...
def calculate_local_rank_and_world_size(
vllm_config: "VllmConfig"
) -> Tuple[int, int]: ...
Import
from lmcache.integration.vllm.utils import (
lmcache_get_or_create_config,
create_lmcache_metadata,
apply_mm_hashes_to_token_ids,
hex_hash_to_int16,
mla_enabled,
extract_mm_features,
get_size_bytes,
get_vllm_torch_dev,
calculate_local_rank_and_world_size,
)
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| vllm_config | VllmConfig | No | Top-level vLLM configuration object (alternative to individual configs) |
| model_config | ModelConfig | No | vLLM model configuration (used if vllm_config is not provided) |
| parallel_config | ParallelConfig | No | vLLM parallel configuration (used if vllm_config is not provided) |
| cache_config | CacheConfig | No | vLLM cache configuration (used if vllm_config is not provided) |
| role | str | No | Role identifier for the LMCache metadata |
| token_ids | torch.Tensor | Yes (for apply_mm_hashes_to_token_ids) | Tensor of token IDs to modify with multimodal hashes |
| mm_hashes | list[str] | Yes (for apply_mm_hashes_to_token_ids) | List of multimodal hash identifier strings |
| mm_positions | list[PlaceholderRange] | Yes (for apply_mm_hashes_to_token_ids) | Corresponding placeholder positions for multimodal tokens |
Outputs
| Name | Type | Description |
|---|---|---|
| config | LMCacheEngineConfig | Singleton LMCache engine configuration loaded from file or environment |
| metadata, config | tuple(LMCacheMetadata, LMCacheEngineConfig) | Metadata and config derived from vLLM configuration |
| int16_hash | int | 16-bit integer hash derived from a multimodal identifier string |
| token_ids | torch.Tensor | Modified token IDs with multimodal placeholders overwritten |
| torch_dev, dev_name | tuple | Torch device module and device name string ("cuda" or "xpu") |
| local_worker_id, local_world_size | Tuple[int, int] | Local rank and local world size for the worker |
Usage Examples
from lmcache.integration.vllm.utils import (
lmcache_get_or_create_config,
create_lmcache_metadata,
hex_hash_to_int16,
)
# Load or create LMCache configuration (thread-safe singleton)
config = lmcache_get_or_create_config()
print(f"Chunk size: {config.chunk_size}")
# Create metadata from vLLM config
metadata, config = create_lmcache_metadata(vllm_config=vllm_config)
# Convert a multimodal hash to a 16-bit integer
hash_val = hex_hash_to_int16("chatcmpl-abc123-image-0")