Workflow:LMCache LMCache CacheBlend KV Reuse
| Knowledge Sources | |
|---|---|
| Domains | LLM_Serving, KV_Cache, Inference_Optimization |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
End-to-end process for reusing KV caches from non-prefix text segments using CacheBlend, which blends precomputed KV caches from arbitrary text chunks into new prompts with RoPE position recovery.
Description
This workflow demonstrates CacheBlend, an advanced KV cache reuse technique that goes beyond simple prefix caching. Traditional prefix caching only reuses KV data when the beginning of a prompt matches exactly. CacheBlend enables reuse of cached KV tensors from arbitrary text segments that appear anywhere in a new prompt, not just at the beginning. It achieves this by recovering rotary positional embeddings (RoPE) to adjust cached KV tensors to their new positions in the combined prompt. This is particularly valuable for RAG workloads where the same document chunks appear in different orders or combinations across prompts.
Usage
Execute this workflow when you have a RAG (Retrieval-Augmented Generation) or document QA workload where the same text chunks appear in different prompts in varying orders or combinations. CacheBlend allows you to precompute KV caches for individual document chunks and reuse them in any prompt that includes those chunks, regardless of their position. Requires a single GPU and works with both CPU and disk storage backends.
Execution Steps
Step 1: Configure LMCache with Blending Enabled
Set up LMCache environment variables or YAML configuration with blending mode enabled. Key parameters include enabling the blend special string delimiter (which marks chunk boundaries in prompts), configuring the recompute ratio (fraction of tokens to recompute for quality recovery), and optionally enabling sparse attention for additional efficiency. Select the storage backend (CPU or disk) for holding precomputed chunk caches.
Key considerations:
- The blend special string serves as a delimiter between independently cached chunks
- The recompute ratio controls the quality-speed tradeoff: higher ratios improve accuracy but reduce speedup
- Layerwise blending processes one transformer layer at a time to minimize memory usage
- Sparse attention is an optional optimization that further reduces recomputation cost
Step 2: Initialize vLLM with CacheBlend Connector
Create a vLLM LLM engine instance with the LMCache KV transfer connector and blending configuration. The connector intercepts KV cache operations and activates the blending code path when chunk boundary delimiters are detected in the prompt. Configure model parameters and GPU memory allocation as with standard LMCache deployment.
Key considerations:
- The same LMCacheConnectorV1 connector handles both prefix caching and blend operations
- Blending is triggered by the presence of the special delimiter string in prompts
- The engine must be configured with sufficient memory for both cached and recomputed KV tensors
- A warmup request is recommended to initialize internal buffers before benchmarking
Step 3: Populate Chunk Caches via Initial Requests
Send initial requests that contain the text chunks you want to cache. As these requests are processed, LMCache stores the KV cache tensors for each chunk delimited by the blend special string. Each chunk's KV cache is stored with its token hash as the key, allowing retrieval regardless of the chunk's position in future prompts.
Key considerations:
- Each unique text chunk needs to be processed at least once to populate its cache
- Chunks can be submitted as part of any prompt; the caching is per-chunk, not per-prompt
- The chunk size configuration determines the granularity of cache storage
- Cached chunks are stored to the configured backend (CPU RAM or disk)
Step 4: Generate with Blended KV Caches
Send new prompts that combine previously cached chunks in different orders or with different surrounding context. CacheBlend detects which chunks have cached KV data, retrieves the cached tensors, applies RoPE position recovery to adjust them to their new positions in the prompt, and selectively recomputes a fraction of tokens (determined by the recompute ratio) to maintain generation quality. The result is significantly faster TTFT compared to computing the full prompt from scratch.
Key considerations:
- Chunks can appear in any order in the new prompt
- RoPE recovery adjusts positional encodings so cached tensors remain valid at new positions
- The recompute ratio determines how many tokens are freshly computed for quality assurance
- Multiple prompts with different chunk combinations all benefit from the same cached data
Step 5: Measure and Validate Performance
Compare generation times between cache-cold requests (no cached data) and cache-warm requests (with blended cached chunks). Verify that the generated output quality is maintained despite using blended caches. The speedup depends on the fraction of the prompt covered by cached chunks and the configured recompute ratio.
Key considerations:
- TTFT reduction is proportional to the fraction of tokens served from cache
- Output quality should be validated against baseline (non-cached) generation
- Lower recompute ratios give better speedup but may affect generation quality for some models
- Disk backends show larger TTFT but support much larger cache capacity