Implementation:LMCache LMCache Async Lookup Client

Knowledge Sources	LMCache
Domains	Cache Lookup, Inter-Process Communication, Asynchronous Processing
Last Updated	2026-02-09 00:00 GMT

Overview

LMCacheAsyncLookupClient and LMCacheAsyncLookupServer provide a ZMQ-based asynchronous lookup mechanism that enables non-blocking KV cache hit detection and prefetching across worker processes.

Description

The LMCacheAsyncLookupClient implements the LookupClientInterface and communicates with one or more LMCacheAsyncLookupServer instances via ZMQ PUSH/PULL sockets. The client hashes token IDs into chunk hashes using a TokenDatabase, sends structured lookup request messages (serialized with msgspec) to all worker servers, and collects responses asynchronously in a background thread. It supports timeout semantics, abort/cleanup of in-flight lookups, and takes the minimum hit count across all workers for consistency in tensor-parallel setups. The server side processes incoming requests by dispatching them to LMCacheEngine.async_lookup_and_prefetch and also handles cleanup messages for aborted requests.

Usage

Use this module when deploying LMCache with vLLM in a multi-worker (tensor parallel or pipeline parallel) environment where asynchronous, non-blocking cache lookups are needed. The client runs in the scheduler process and the servers run in each worker process.

Code Reference

Source Location

• Repository: LMCache
• File: lmcache/v1/lookup_client/lmcache_async_lookup_client.py
• Lines: 1-407

Signature

class LMCacheAsyncLookupClient(LookupClientInterface):
    def __init__(self, config: LMCacheEngineConfig, metadata: LMCacheMetadata) -> None: ...
    def lookup_cache(self, lookup_id: str) -> Optional[int]: ...
    def lookup(self, token_ids: Union[torch.Tensor, list[int]], lookup_id: str,
               request_configs: Optional[dict] = None) -> Optional[int]: ...
    def process_responses_from_workers(self) -> None: ...
    def clear_lookup_status(self, lookup_id: str) -> None: ...
    def cancel_lookup(self, lookup_id: str) -> None: ...
    def supports_producer_reuse(self) -> bool: ...
    def close(self) -> None: ...

class LMCacheAsyncLookupServer:
    def __init__(self, lmcache_engine: LMCacheEngine, metadata: LMCacheMetadata) -> None: ...
    def process_requests_from_scheduler(self) -> None: ...
    def send_response_to_scheduler(self, lookup_id: str, num_hit_tokens: int) -> None: ...
    def close(self) -> None: ...

Import

from lmcache.v1.lookup_client.lmcache_async_lookup_client import (
    LMCacheAsyncLookupClient,
    LMCacheAsyncLookupServer,
)

I/O Contract

Inputs

Name	Type	Required	Description
config	LMCacheEngineConfig	Yes	Engine configuration including lookup timeout and extra config settings
metadata	LMCacheMetadata	Yes	Metadata including engine_id, world_size, worker_id, and RPC port
token_ids	Union[torch.Tensor, list[int]]	Yes (for lookup)	Token IDs to hash and look up in the cache
lookup_id	str	Yes	Unique identifier for the lookup request (typically the request ID)
request_configs	Optional[dict]	No	Optional per-request configuration parameters

Outputs

Name	Type	Description
lookup_cache()	Optional[int]	-1 if not found, None if ongoing, int >= 0 for number of hit tokens
lookup()	Optional[int]	Always returns None (async; results arrive via background thread)
supports_producer_reuse()	bool	Always returns True, indicating support for producer KV cache reuse

Usage Examples

from lmcache.v1.lookup_client.lmcache_async_lookup_client import (
    LMCacheAsyncLookupClient,
    LMCacheAsyncLookupServer,
)

# Client side (scheduler process)
client = LMCacheAsyncLookupClient(config, metadata)
client.lookup(token_ids, lookup_id="req-123")

# Poll for result
result = client.lookup_cache("req-123")
# result is None while ongoing, int >= 0 when complete

# Cleanup
client.clear_lookup_status("req-123")
client.close()

# Server side (worker process)
server = LMCacheAsyncLookupServer(lmcache_engine, metadata)
# Server processes requests in background thread
server.send_response_to_scheduler("req-123", num_hit_tokens=512)
server.close()