Implementation:LMCache LMCache L1 Memory Manager

Knowledge Sources	LMCache
Domains	Memory Management, Distributed Storage
Last Updated	2026-02-09 00:00 GMT

Overview

Manages allocation, deallocation, and memory usage tracking for the L1 tier of the distributed storage system.

Description

The L1MemoryManager class wraps a memory allocator (either LazyMemoryAllocator for lazy/on-demand pinned memory allocation or MixedMemoryAllocator for eager allocation) and provides thread-safe batch allocation and deallocation of MemoryObj instances based on MemoryLayoutDesc specifications. It supports querying memory usage (used bytes vs. total bytes) for eviction decisions, exposing the underlying virtual memory space for RDMA communication, and performing memory integrity checks. The factory function create_memory_allocator selects the appropriate allocator based on the L1MemoryManagerConfig.

Usage

Use L1MemoryManager within the distributed storage manager's L1 tier to allocate memory for KV cache chunks before writing and to free memory after eviction or cleanup. Query get_memory_usage to inform eviction controller decisions. Access get_vm_space when setting up RDMA communication buffers.

Code Reference

Source Location

• Repository: LMCache
• File: lmcache/v1/distributed/memory_manager.py
• Lines: 1-163

Signature

def create_memory_allocator(config: L1MemoryManagerConfig) -> MemoryAllocatorInterface: ...

class L1MemoryManager:
    def __init__(self, config: L1MemoryManagerConfig) -> None: ...
    def allocate(self, layout_desc: MemoryLayoutDesc, count: int) -> tuple[L1Error, list[MemoryObj]]: ...
    def free(self, mem_objs: list[MemoryObj]) -> L1Error: ...
    def get_vm_space(self) -> torch.Tensor: ...
    def get_memory_usage(self) -> tuple[int, int]: ...
    def close(self) -> None: ...
    def memcheck(self) -> None: ...

Import

from lmcache.v1.distributed.memory_manager import L1MemoryManager, create_memory_allocator

I/O Contract

Inputs

Name	Type	Required	Description
config	L1MemoryManagerConfig	Yes	Configuration specifying memory size, lazy mode, init size, and alignment
layout_desc	MemoryLayoutDesc	Yes	Tensor shapes and dtypes for memory allocation
count	int	Yes	Number of memory objects to allocate in a batch
mem_objs	list[MemoryObj]	Yes	Memory objects to free (for the free method)

Outputs

Name	Type	Description
(error, objects)	tuple[L1Error, list[MemoryObj]]	Error code (SUCCESS or OUT_OF_MEMORY) and list of allocated memory objects
error	L1Error	Error code from free operation (SUCCESS on success)
vm_space	torch.Tensor	Underlying virtual memory space tensor for RDMA registration
(used, total)	tuple[int, int]	Used memory in bytes and total memory in bytes

Usage Examples

from lmcache.v1.distributed.memory_manager import L1MemoryManager
from lmcache.v1.distributed.config import L1MemoryManagerConfig
from lmcache.v1.distributed.api import MemoryLayoutDesc
from lmcache.v1.distributed.error import L1Error
import torch

config = L1MemoryManagerConfig(
    size_in_bytes=64 * (1 << 30),  # 64 GB
    use_lazy=True,
)
manager = L1MemoryManager(config)

# Allocate memory for 10 chunks
layout = MemoryLayoutDesc(
    shapes=[torch.Size([2, 32, 128])],
    dtypes=[torch.float16],
)
error, mem_objs = manager.allocate(layout, count=10)
if error == L1Error.SUCCESS:
    # Use mem_objs for KV cache storage
    pass

# Check memory usage
used, total = manager.get_memory_usage()
print(f"Memory usage: {used / (1 << 30):.1f} GB / {total / (1 << 30):.1f} GB")

# Free memory
manager.free(mem_objs)
manager.close()