Principle:Ollama Ollama KVCache Composite Caching
| Knowledge Sources | |
|---|---|
| Domains | KV Cache, Hybrid Memory |
| Last Updated | 2025-02-15 00:00 GMT |
Overview
Composite Caching is the principle of combining multiple distinct caching strategies into a unified cache system to support hybrid model architectures that use different attention or memory patterns across their layers. This approach enables models that mix standard attention with sliding window attention, recurrent state, or other memory mechanisms to be served through a single cache interface.
Core Concepts
Hybrid Model Architectures
Modern LLM architectures increasingly combine different layer types within a single model. Examples include models that alternate between full causal attention layers and sliding window attention layers (Interleaved Sliding Window Attention, or ISWA), models that combine transformer attention layers with recurrent layers (Mamba, RWKV, Griffin), and models that use different attention patterns at different depths. Each layer type requires a different caching strategy: full attention needs unbounded (up to context length) KV storage, sliding window needs fixed-size circular buffers, and recurrent layers need compact fixed-size hidden state storage.
Cache Composition Pattern
The composite cache pattern wraps two or more cache implementations behind a single interface. The composite routes operations to the appropriate sub-cache based on the layer index: layers using full attention are directed to a causal KV cache, layers using sliding window attention are directed to a windowed cache, and recurrent layers are directed to a state cache. This composition is transparent to the inference engine, which interacts with a single cache object regardless of the underlying heterogeneity.
Sliding Window Attention Cache
Sliding window (or local) attention restricts each token to attend only to the most recent W tokens, where W is the window size. The cache for sliding window layers only needs to store the last W key-value pairs, using a ring buffer that overwrites the oldest entries as new tokens are generated. This provides a constant memory footprint per layer regardless of total sequence length, dramatically reducing memory usage for long sequences while maintaining the model's designed attention pattern.
Recurrent State Cache
Recurrent layers (such as Mamba's selective state space layers or RWKV's linear attention layers) maintain a fixed-size hidden state that is updated at each timestep rather than an ever-growing key-value cache. The state cache stores these fixed-size tensors, which are overwritten (not appended) at each step. This fundamentally different memory access pattern requires its own cache implementation: allocation is O(1) per layer regardless of sequence length, update is an in-place write, and there is no concept of cache growth or eviction.
Coordinated Lifecycle Management
The composite cache must coordinate lifecycle operations across all sub-caches. Creating a new sequence allocates entries in all sub-caches. Removing a sequence frees entries in all sub-caches. Defragmentation must be coordinated so that logical sequence identifiers remain consistent across sub-caches. Memory budget allocation must be partitioned between sub-caches based on their different scaling characteristics (full attention cache grows with sequence length, sliding window cache is constant, recurrent state is constant).
Implementation Notes
In the Ollama codebase, composite caching is implemented to support hybrid architectures such as models using Interleaved Sliding Window Attention (ISWA) and models combining attention with recurrent layers. The ISWA cache composes two causal KV caches with different maximum context lengths: one for full-attention layers (with the model's full context window) and one for sliding-window layers (with a smaller fixed window). The hybrid cache composes a causal KV cache for attention layers with a recurrent state cache for Mamba-style layers. Both composite implementations expose the standard cache interface, allowing the inference engine to operate uniformly across pure-attention, ISWA, and hybrid-recurrent models.