Implementation:LMCache LMCache EIC Connector

Knowledge Sources	LMCache
Domains	Caching, Storage Connectors, High Performance Computing
Last Updated	2026-02-09 00:00 GMT

Overview

EICConnector is a remote storage connector that uses the EIC (Elastic Intelligent Cache) client library for high-performance KV cache storage with optional GDR (GPU Direct RDMA) transport support.

Description

The EICConnector connects to an EIC-based caching service for storing and retrieving LLM KV cache data. It reads its configuration from a YAML file specified by the LMCACHE_CONFIG_FILE environment variable, supporting parameters such as instance ID, thread count, log directory, transport type, TTL, and namespace. The connector supports three transport modes including standard and GDR (GPU Direct RDMA), and when GDR is active, it registers pinned memory buffers with the EIC client for zero-copy transfers. All operations are submitted through an AsyncPQExecutor priority queue with four priority levels (PEEK, PREFETCH, GET, PUT). Metadata and data are stored as separate keys with the metadata key suffixed by _meta. A PerformanceTimer utility class provides fine-grained timing instrumentation for each operation step. The connector pre-builds 2048 connections at initialization time for connection warming.

Usage

Use EICConnector when deploying LMCache with an EIC-compatible caching infrastructure, particularly in environments that support GPU Direct RDMA for maximum transfer performance. The remote URL must start with eic://.

Code Reference

Source Location

• Repository: LMCache
• File: lmcache/v1/storage_backend/connector/eic_connector.py
• Lines: 1-746

Signature

class Priorities(IntEnum):
    PEEK = auto()
    PREFETCH = auto()
    GET = auto()
    PUT = auto()

class PerformanceTimer:
    def __init__(self, key, op): ...
    def set_size(self, size): ...
    def start(self, operation_name): ...
    def stop(self, operation_name): ...

class EICConnector(RemoteConnector):
    def __init__(self, endpoint: str, loop: asyncio.AbstractEventLoop,
                 memory_allocator: LocalCPUBackend): ...
    def prebuilt_connection(self) -> None: ...
    async def exists(self, key: CacheEngineKey) -> bool: ...
    def exists_sync(self, key: CacheEngineKey) -> bool: ...
    async def get(self, key: CacheEngineKey) -> Optional[MemoryObj]: ...
    async def get_meta(self, key_str: str) -> Optional[RemoteMetadata]: ...
    async def get_data(self, key_str: str, meta: RemoteMetadata) -> Optional[MemoryObj]: ...
    async def put(self, key: CacheEngineKey, memory_obj: MemoryObj): ...
    async def batched_put(self, keys: List[CacheEngineKey], memory_objs: List[MemoryObj]): ...
    async def batched_get(self, keys: List[CacheEngineKey]) -> List[Optional[MemoryObj]]: ...
    async def batched_async_contains(self, lookup_id: str, keys: List[CacheEngineKey], pin: bool = False) -> int: ...
    async def batched_get_non_blocking(self, lookup_id: str, keys: List[CacheEngineKey]) -> List[MemoryObj]: ...
    async def close(self): ...
    async def list(self) -> List[str]: ...

Import

from lmcache.v1.storage_backend.connector.eic_connector import EICConnector

I/O Contract

Inputs

Name	Type	Required	Description
endpoint	str	Yes	EIC service endpoint URL (must start with "eic://")
loop	asyncio.AbstractEventLoop	Yes	Asyncio event loop for async operations
memory_allocator	LocalCPUBackend	Yes	CPU backend used for memory allocation and providing config/metadata

Outputs

Name	Type	Description
EICConnector	RemoteConnector	An initialized connector with EIC client connection, priority queue executor, and optional GDR memory registration

Usage Examples

from lmcache.v1.storage_backend.connector.eic_connector import EICConnector

# EICConnector is typically created via the CreateConnector factory
# with a remote_url starting with "eic://"
connector = EICConnector(
    endpoint="eic://10.0.0.1:8080",
    loop=asyncio.get_event_loop(),
    memory_allocator=local_cpu_backend,
)

# Check key existence
exists = await connector.exists(cache_key)

# Retrieve KV cache
memory_obj = await connector.get(cache_key)

# Store KV cache
await connector.put(cache_key, memory_obj)

# Batch operations
results = await connector.batched_get(keys)