Implementation:LMCache LMCache GDS Backend

Knowledge Sources	LMCache
Domains	Caching, GPU Direct Storage, High Performance I/O
Last Updated	2026-02-09 00:00 GMT

Overview

GdsBackend is a storage backend that leverages NVIDIA's cuFile API for GPU Direct Storage (GDS) to perform direct DMA transfers between GPU memory and GDS-supported filesystems, bypassing CPU-mediated copies.

Description

The GdsBackend implements the AllocatorBackendInterface and provides persistent KV cache storage on disk with GPU-optimized I/O. It uses a two-level directory hierarchy (/{gds_path}/{2-char-hash}/{2-char-hash}/) for efficient filesystem scanning and parallel metadata loading at startup. Each cache entry is stored as two files: a data file (with .kvcache.safetensors or .weka1 suffix depending on the filesystem) containing a 4KB metadata header followed by raw tensor data, and a separate metadata file. The backend auto-detects the filesystem type (wekafs, tmpfs, overlayfs, etc.) and adapts its behavior accordingly: on WekaFS it enforces cuFile usage and activates a thread pool for batched I/O; on tmpfs/overlayfs it disables cuFile and falls back to cudaMemcpy with mmap. When cuFile is unavailable, it uses mmap-based POSIX I/O with cudaMemcpy for GPU-to-host and host-to-GPU transfers. The backend maintains a hot cache (OrderedDict) of metadata for fast key lookups without disk reads, scanned at initialization time. It uses a CuFileMemoryAllocator for GPU memory management with optional base pointer support to avoid bounce buffers. Write operations are performed asynchronously with atomic rename for crash safety.

Usage

Use GdsBackend when deploying LMCache with NVMe SSDs or network filesystems that support GPU Direct Storage for maximum I/O throughput between GPU memory and persistent storage. Requires gds_path and cufile_buffer_size to be configured in the LMCache engine config.

Code Reference

Source Location

• Repository: LMCache
• File: lmcache/v1/storage_backend/gds_backend.py
• Lines: 1-919

Signature

def pack_metadata(tensor, fmt: MemoryFormat, **extra_metadata) -> bytes: ...
def unpack_metadata(buffer: bytes): ...
def get_fstype(path): ...

class GdsBackend(AllocatorBackendInterface):
    def __init__(
        self,
        config: LMCacheEngineConfig,
        metadata: LMCacheMetadata,
        loop: asyncio.AbstractEventLoop,
        dst_device: str = "cuda",
    ): ...
    def contains(self, key: CacheEngineKey, pin: bool = False) -> bool: ...
    def submit_put_task(
        self, key: CacheEngineKey, memory_obj: MemoryObj,
        on_complete_callback: Optional[Callable[[CacheEngineKey], None]] = None,
    ) -> Future: ...
    def batched_submit_put_task(
        self, keys: Sequence[CacheEngineKey], memory_objs: List[MemoryObj],
        transfer_spec: Any = None,
        on_complete_callback: Optional[Callable[[CacheEngineKey], None]] = None,
    ) -> Union[List[Future], None]: ...
    def get_blocking(self, key: CacheEngineKey) -> Optional[MemoryObj]: ...
    def batched_get_blocking(self, keys: List[CacheEngineKey]) -> List[Optional[MemoryObj]]: ...
    def allocate(
        self, shapes, dtypes, fmt=MemoryFormat.KV_2LTD,
        eviction: bool = True, busy_loop: bool = True,
    ) -> Optional[MemoryObj]: ...
    def batched_allocate(
        self, shapes, dtypes, batch_size: int, fmt=MemoryFormat.KV_2LTD,
        eviction: bool = True, busy_loop: bool = True,
    ) -> Optional[list[MemoryObj]]: ...
    def pin(self, key: CacheEngineKey) -> bool: ...
    def unpin(self, key: CacheEngineKey) -> bool: ...
    def remove(self, key: CacheEngineKey, force: bool = True): ...
    def close(self) -> None: ...

Import

from lmcache.v1.storage_backend.gds_backend import GdsBackend

I/O Contract

Inputs

Name	Type	Required	Description
config	LMCacheEngineConfig	Yes	Engine config with gds_path, cufile_buffer_size, and optional extra_config (use_cufile, use_direct_io, gds_io_threads)
metadata	LMCacheMetadata	Yes	LMCache metadata describing tensor shapes, dtypes, and chunk size
loop	asyncio.AbstractEventLoop	Yes	Asyncio event loop for async disk I/O operations
dst_device	str	No	Target CUDA device string (default: "cuda")

Outputs

Name	Type	Description
GdsBackend	AllocatorBackendInterface	A GPU Direct Storage backend with CuFileMemoryAllocator, hot cache metadata index, and filesystem-aware I/O strategy

Usage Examples

from lmcache.v1.storage_backend.gds_backend import GdsBackend

# Initialize GDS backend
gds_backend = GdsBackend(
    config=lmcache_config,       # Must have gds_path and cufile_buffer_size set
    metadata=lmcache_metadata,
    loop=asyncio.get_event_loop(),
    dst_device="cuda:0",
)

# Check if key exists in persistent cache
if gds_backend.contains(cache_key):
    # Load KV cache from disk directly to GPU
    memory_obj = gds_backend.get_blocking(cache_key)

# Async write KV cache from GPU to disk
future = gds_backend.submit_put_task(cache_key, memory_obj)

# Batched read with thread pool (WekaFS)
results = gds_backend.batched_get_blocking(keys)

# Allocate GPU memory for new cache entries
memory_obj = gds_backend.allocate(shapes, dtypes)

# Cleanup
gds_backend.close()