Implementation:Ollama Ollama Llama Memory Hybrid

Knowledge Sources	Ollama
Domains	LLM Inference, Memory Management
Last Updated	2025-02-15 00:00 GMT

Overview

Implements the hybrid memory system that combines a KV cache (for attention layers) with recurrent state memory (for SSM/Mamba layers) in a single unified interface.

Description

The constructor creates a llama_kv_cache for attention-based layers and a llama_memory_recurrent for recurrent layers, using layer filter callbacks based on hparams.is_recurrent(il) to route each layer to the appropriate memory type. All llama_memory_i interface methods delegate to both memory instances. The llama_memory_hybrid_context class manages the combined context during inference, providing access to both sub-contexts.

Usage

Used for hybrid architectures like Jamba and Granite Hybrid that interleave attention layers with recurrent (Mamba/SSM) layers within a single model. Without this, such architectures could not be served.

Code Reference

Source Location

• Repository: Ollama
• File: llama/llama.cpp/src/llama-memory-hybrid.cpp
• Lines: 1-268

Signature

llama_memory_hybrid::llama_memory_hybrid(
        const llama_model & model,
                ggml_type   type_k, ggml_type type_v,
                     bool   v_trans,
                 uint32_t   kv_size, uint32_t n_pad, uint32_t n_swa,
           llama_swa_type   swa_type,
                ggml_type   type_r, ggml_type type_s,
                 uint32_t   rs_size,
                 uint32_t   n_seq_max,
                     bool   offload, bool unified,
    const layer_filter_cb & filter_attn,
    const layer_filter_cb & filter_recr);

llama_kv_cache * get_mem_attn() const;
llama_memory_recurrent * get_mem_recr() const;

Import

#include "llama-memory-hybrid.h"

I/O Contract

Inputs

Name	Type	Required	Description
model	const llama_model &	Yes	Model with mixed layer types
type_k, type_v	ggml_type	Yes	KV cache data types for attention layers
type_r, type_s	ggml_type	Yes	State data types for recurrent layers
kv_size	uint32_t	Yes	KV cache size for attention layers
rs_size	uint32_t	Yes	Recurrent state size

Outputs

Name	Type	Description
get_mem_attn()	llama_kv_cache*	The attention KV cache
get_mem_recr()	llama_memory_recurrent*	The recurrent state memory

Usage Examples

// Created internally for hybrid models (Jamba, Granite Hybrid)
// Access sub-contexts during graph building:
const auto * hybrid_ctx = static_cast<const llama_memory_hybrid_context *>(mctx);
const auto * attn_ctx = hybrid_ctx->get_attn();
const auto * recr_ctx = hybrid_ctx->get_recr();

Related Pages

• Principle:Ollama_Ollama_LLM_Memory_Architecture