Implementation:LMCache LMCache SageMaker HyperPod Connector

Knowledge Sources	LMCache
Domains	Caching, Storage Connectors, AWS SageMaker, Shared Memory
Last Updated	2026-02-09 00:00 GMT

Overview

SageMakerHyperPodConnector is a high-performance remote connector that communicates with a SageMaker HyperPod KV cache daemon using shared memory for zero-copy reads and HTTP for control plane operations and writes.

Description

The SageMakerHyperPodConnector implements a lease-based protocol for accessing KV cache data stored in a SageMaker HyperPod service. The data plane uses shared memory segments for zero-copy read access: when retrieving data, the connector acquires a lease from the daemon via HTTP, reads directly from shared memory using the lease's offset information, and then immediately releases the lease. The control plane uses HTTP with connection pooling (via aiohttp) for lease acquisition, release, and PUT operations. PUT operations use HTTP streaming with configurable chunk sizes (default 64KB). The connector supports configurable concurrency limits, connection pools, request timeouts, lease TTLs, and maximum lease sizes. It includes built-in statistics tracking for get/put success/failure counts and lease operations, integrated with the LMCache observability system (LMCStatsMonitor). All operations are routed through an AsyncPQExecutor with three priority levels: LEASE (highest), PREFETCH, and PUT.

Usage

Use SageMakerHyperPodConnector when deploying LMCache on AWS SageMaker HyperPod infrastructure with the ai-toolkit daemon running. The daemon manages a shared memory segment for zero-copy data access and handles cache eviction through the lease mechanism. The remote URL should point to the ai-toolkit daemon's HTTP endpoint.

Code Reference

Source Location

• Repository: LMCache
• File: lmcache/v1/storage_backend/connector/sagemaker_hyperpod_connector.py
• Lines: 1-987

Signature

class Priorities(IntEnum):
    LEASE = 0
    PREFETCH = auto()
    PUT = auto()

@dataclass
class LeaseInfo:
    lease_id: str
    offsets: List[Tuple[int, int]]

class SageMakerHyperPodConnector(RemoteConnector):
    def __init__(
        self,
        sagemaker_hyperpod_url: str,
        loop: asyncio.AbstractEventLoop,
        local_cpu_backend: LocalCPUBackend,
        bucket_name: str,
        shared_memory_name: Optional[str],
        max_concurrent_requests: int,
        max_connections: int,
        max_connections_per_host: int,
        timeout_ms: int,
        lease_ttl_s: float = 10.0,
        put_stream_chunk_bytes: int = 65536,
        max_lease_size_mb: Optional[float] = None,
        **kwargs,
    ): ...
    async def exists(self, key: CacheEngineKey) -> bool: ...
    def exists_sync(self, key: CacheEngineKey) -> bool: ...
    async def get(self, key: CacheEngineKey) -> Optional[MemoryObj]: ...
    async def put(self, key: CacheEngineKey, memory_obj: MemoryObj): ...
    async def batched_get(self, keys: List[CacheEngineKey]) -> List[Optional[MemoryObj]]: ...
    async def batched_put(self, keys: List[CacheEngineKey], memory_objs: List[MemoryObj]): ...
    async def batched_async_contains(self, lookup_id: str, keys: List[CacheEngineKey], pin: bool = False) -> int: ...
    async def batched_get_non_blocking(self, lookup_id: str, keys: List[CacheEngineKey]) -> List[MemoryObj]: ...
    async def list(self) -> List[str]: ...
    def remove_sync(self, key: CacheEngineKey) -> bool: ...
    async def close(self): ...

Import

from lmcache.v1.storage_backend.connector.sagemaker_hyperpod_connector import (
    SageMakerHyperPodConnector,
)

I/O Contract

Inputs

Name	Type	Required	Description
sagemaker_hyperpod_url	str	Yes	Base URL of the ai-toolkit daemon HTTP endpoint
loop	asyncio.AbstractEventLoop	Yes	Asyncio event loop for async operations
local_cpu_backend	LocalCPUBackend	Yes	CPU backend for local memory allocation
bucket_name	str	Yes	Bucket name for KV storage namespace
shared_memory_name	Optional[str]	No	Name of the POSIX shared memory segment (None disables shared memory reads)
max_concurrent_requests	int	Yes	Maximum number of concurrent HTTP requests for control/data planes
max_connections	int	Yes	Maximum total HTTP connections in the connection pool
max_connections_per_host	int	Yes	Maximum HTTP connections per host
timeout_ms	int	Yes	Timeout in milliseconds for lease acquisition requests
lease_ttl_s	float	No	Server-side lease timeout in seconds (default: 10.0)
put_stream_chunk_bytes	int	No	Chunk size in bytes for streaming PUT requests (default: 65536)
max_lease_size_mb	Optional[float]	No	Maximum lease size in MB; leases exceeding this are immediately released

Outputs

Name	Type	Description
SageMakerHyperPodConnector	RemoteConnector	A connector with shared memory data plane and HTTP control plane, integrated with LMCache observability

Usage Examples

from lmcache.v1.storage_backend.connector.sagemaker_hyperpod_connector import (
    SageMakerHyperPodConnector,
)

connector = SageMakerHyperPodConnector(
    sagemaker_hyperpod_url="http://localhost:8080",
    loop=asyncio.get_event_loop(),
    local_cpu_backend=local_cpu_backend,
    bucket_name="my-kv-cache",
    shared_memory_name="lmcache_shm",
    max_concurrent_requests=64,
    max_connections=128,
    max_connections_per_host=64,
    timeout_ms=5000,
)

# Retrieve data (lease acquire -> shared memory read -> lease release)
memory_obj = await connector.get(cache_key)

# Store data (HTTP streaming PUT)
await connector.put(cache_key, memory_obj)

# Check existence (lease-based)
exists = await connector.exists(cache_key)

# Cleanup
await connector.close()