Principle:Vllm project Vllm LoRA Engine Configuration
| Knowledge Sources | |
|---|---|
| Domains | LLM Serving, Model Adaptation, Engine Configuration |
| Last Updated | 2026-02-08 13:00 GMT |
Overview
LoRA engine configuration is the process of initializing an LLM inference engine with the parameters required to support dynamic loading and serving of LoRA adapters.
Description
To serve LoRA adapters alongside a base model, the inference engine must be explicitly configured with LoRA support enabled. This involves setting several parameters that control the engine's behavior regarding adapter management: how many adapters can be active simultaneously in a single batch, the maximum rank of adapters the engine can handle, and how many adapters can be cached in CPU memory for fast swapping.
These configuration choices have direct implications for memory allocation, throughput, and latency. The engine pre-allocates GPU memory for LoRA adapter slots based on the configured maximum number of concurrent adapters and maximum rank. Higher values increase memory consumption but allow more adapters to be served concurrently without swapping.
Usage
Use LoRA engine configuration when:
- Setting up a vLLM engine to serve one or more LoRA adapters alongside a base model
- Tuning the trade-off between memory usage and concurrent adapter capacity
- Configuring a production deployment that must handle requests targeting different fine-tuned adapters
- Optimizing for workloads where the number of distinct adapters exceeds the GPU memory budget (using CPU LoRA caching)
Theoretical Basis
The engine configuration for LoRA serving centers on three resource dimensions:
GPU Adapter Slots (max_loras): Each GPU adapter slot requires pre-allocated memory proportional to the maximum LoRA rank. The engine maintains a fixed number of these slots, and only requests whose adapters occupy a slot can be processed in the current batch. When the number of distinct adapters in a batch exceeds max_loras, the engine must schedule adapter swaps across batches.
Maximum Rank (max_lora_rank): The rank determines the size of the low-rank matrices (B and A) for each adapted layer. Higher ranks produce larger adapter weight matrices and require more GPU memory per slot. Setting this to the actual rank of the adapters being served avoids wasting memory on unused capacity. Supported values are powers of 2 from 1 to 512, plus 320.
CPU Cache (max_cpu_loras): Adapters not currently occupying a GPU slot can be cached in CPU memory for fast re-loading. This is critical for workloads with many distinct adapters, as it avoids re-reading adapter weights from disk. The CPU cache size must be at least as large as max_loras.
Sharding (fully_sharded_loras): By default, only half of the LoRA computation is sharded across tensor-parallel ranks. Enabling fully sharded LoRAs distributes all LoRA computation, which can improve performance at high sequence lengths, high ranks, or high tensor-parallel sizes.