Implementation:Ollama Ollama Llama Memory Interface

Knowledge Sources	Ollama
Domains	LLM Inference, Memory Management
Last Updated	2025-02-15 00:00 GMT

Overview

Header declaring the abstract memory interface (llama_memory_i) and memory context interface (llama_memory_context_i) that all memory implementations must conform to.

Description

Defines llama_memory_params with KV cache type configuration and llama_memory_status enum (SUCCESS, NO_UPDATE, FAILED_PREPARE, FAILED_COMPUTE). llama_memory_context_i is the per-batch context interface with next(), apply(), get_ubatch(), and get_status() methods. llama_memory_i is the main memory interface with methods for batch processing setup, sequence operations, position queries, memory breakdown, and state serialization.

Usage

This is the fundamental abstraction that allows different memory backends (KV cache, recurrent state, ISWA, hybrid) to be used interchangeably through the same inference pipeline.

Code Reference

Source Location

• Repository: Ollama
• File: llama/llama.cpp/src/llama-memory.h
• Lines: 1-122

Signature

enum llama_memory_status {
    LLAMA_MEMORY_STATUS_SUCCESS = 0,
    LLAMA_MEMORY_STATUS_NO_UPDATE,
    LLAMA_MEMORY_STATUS_FAILED_PREPARE,
    LLAMA_MEMORY_STATUS_FAILED_COMPUTE,
};

struct llama_memory_context_i {
    virtual ~llama_memory_context_i() = default;
    virtual bool next() = 0;
    virtual bool apply() = 0;
    virtual const llama_ubatch & get_ubatch() const = 0;
    virtual llama_memory_status get_status() const = 0;
};

struct llama_memory_i {
    using layer_filter_cb = std::function<bool(int32_t il)>;
    using layer_reuse_cb = std::function<int32_t(int32_t il)>;

    virtual llama_memory_context_ptr init_batch(llama_batch_allocr & balloc, uint32_t n_ubatch, bool embd_all) = 0;
    virtual llama_memory_context_ptr init_full() = 0;
    virtual llama_memory_context_ptr init_update(llama_context * lctx, bool optimize) = 0;
    virtual bool get_can_shift() const = 0;
    virtual void clear(bool data) = 0;
    virtual bool seq_rm(llama_seq_id seq_id, llama_pos p0, llama_pos p1) = 0;
    virtual void seq_cp(llama_seq_id seq_id_src, llama_seq_id seq_id_dst, llama_pos p0, llama_pos p1) = 0;
};

using llama_memory_ptr = std::unique_ptr<llama_memory_i>;

Import

#include "llama-memory.h"

I/O Contract

Inputs

Name	Type	Required	Description
balloc	llama_batch_allocr &	Yes	Batch allocator for splitting into ubatches
n_ubatch	uint32_t	Yes	Maximum ubatch size
embd_all	bool	Yes	Whether all tokens produce embeddings

Outputs

Name	Type	Description
llama_memory_context_ptr	unique_ptr	Context for processing ubatches
llama_memory_status	enum	Status of the memory operation

Usage Examples

#include "llama-memory.h"

// The memory interface is used polymorphically:
llama_memory_i * memory = model.memory.get();

// Initialize for batch processing
auto ctx = memory->init_batch(balloc, n_ubatch, embd_all);
if (ctx->get_status() == LLAMA_MEMORY_STATUS_SUCCESS) {
    do {
        ctx->apply();
        // ... process ubatch ...
    } while (ctx->next());
}

Related Pages

• Principle:Ollama_Ollama_LLM_Memory_Architecture