Implementation:LMCache LMCache Bypass Lookup Client

Knowledge Sources	LMCache
Domains	KV Cache, Distributed Inference
Last Updated	2026-02-09 00:00 GMT

Overview

LMCacheBypassLookupClient is a lookup client that directly invokes the LMCacheEngine lookup method in-process, bypassing ZMQ-based inter-process communication.

Description

The bypass lookup client implements the LookupClientInterface and is designed for scenarios such as MLA (Multi-head Latent Attention), where only rank 0 needs to perform lookups and there is no need for cross-process messaging overhead. It processes token IDs through a token database to compute chunk hashes and offsets, then delegates the actual lookup to the provided LMCacheEngine instance. In blending mode, tokens are passed directly to the engine without prior hash computation. All exceptions are caught and logged, returning 0 to indicate no cache match.

Usage

Use this client when the lookup can be performed within the same process as the cache engine, particularly in MLA configurations where only the first rank participates in lookups. This avoids the latency and complexity of ZMQ-based RPC communication.

Code Reference

Source Location

• Repository: LMCache
• File: lmcache/v1/lookup_client/lmcache_lookup_client_bypass.py
• Lines: 1-99

Signature

class LMCacheBypassLookupClient(LookupClientInterface):
    def __init__(
        self,
        config: LMCacheEngineConfig,
        metadata: LMCacheMetadata,
        lmcache_engine: LMCacheEngine,
    ): ...

    def lookup(
        self,
        token_ids: Union[torch.Tensor, list[int]],
        lookup_id: str,
        request_configs: Optional[dict] = None,
    ) -> Optional[int]: ...

    def supports_producer_reuse(self) -> bool: ...

    def close(self): ...

Import

from lmcache.v1.lookup_client.lmcache_lookup_client_bypass import LMCacheBypassLookupClient

I/O Contract

Inputs

Name	Type	Required	Description
config	LMCacheEngineConfig	Yes	The LMCache engine configuration object
metadata	LMCacheMetadata	Yes	Metadata extracted from the serving engine
lmcache_engine	LMCacheEngine	Yes	The engine instance used for direct lookup calls
token_ids	Union[torch.Tensor, list[int]]	Yes	Token IDs to look up in the cache
lookup_id	str	Yes	Unique identifier for this lookup request
request_configs	Optional[dict]	No	Additional per-request configuration options

Outputs

Name	Type	Description
lookup result	Optional[int]	Number of tokens matched in cache, or 0 if no match or an error occurred
supports_producer_reuse	bool	Always returns True, indicating support for producer KV cache reuse

Usage Examples

from lmcache.v1.lookup_client.lmcache_lookup_client_bypass import LMCacheBypassLookupClient

# Initialize with an existing engine
client = LMCacheBypassLookupClient(config, metadata, lmcache_engine)

# Perform a lookup
matched_tokens = client.lookup(token_ids=[1, 2, 3, 4], lookup_id="req-001")

# Clean up
client.close()