Implementation:Ollama Ollama Llama Graph Header

Knowledge Sources	Ollama
Domains	Compute Graph, Inference
Last Updated	2025-02-15 00:00 GMT

Overview

Header declaring the compute graph construction framework, including graph types, input node classes, FFN/normalization enums, the graph result container, and the main llm_graph_context builder class.

Description

Defines enums for graph types (default, encoder, decoder), FFN activation functions (SiLU, GELU, ReLU, SwiGLU, GeGLU, ReGLU variants), FFN gate types (sequential, parallel), and normalization types (LayerNorm, RMSNorm, GroupNorm). Declares llm_graph_input_i as the base class for input nodes, with specialized subclasses for embeddings, positions, attention masks, output indices, pooling, cross-attention, recurrent state, and hybrid memory. llm_graph_result holds the built graph, its inputs, and output tensors. llm_graph_context provides the builder API with methods for constructing normalization, FFN, attention (no-cache, KV, ISWA, cross), MoE, pooling, and other graph components. Also defines llama_cross for encoder-decoder cross-attention data transfer.

Usage

Include this header when implementing a new model architecture or working with the graph construction framework. Every model architecture's build_graph method uses the llm_graph_context builder.

Code Reference

Source Location

• Repository: Ollama
• File: llama/llama.cpp/src/llama-graph.h
• Lines: 1-845

Signature

enum llm_graph_type { LLM_GRAPH_TYPE_DEFAULT, LLM_GRAPH_TYPE_ENCODER, LLM_GRAPH_TYPE_DECODER };
enum llm_ffn_op_type { LLM_FFN_SILU, LLM_FFN_GELU, LLM_FFN_RELU, LLM_FFN_SWIGLU, LLM_FFN_GEGLU };
enum llm_norm_type { LLM_NORM, LLM_NORM_RMS, LLM_NORM_GROUP };

class llm_graph_input_i {
public:
    virtual void set_input(const llama_ubatch * ubatch) = 0;
    virtual bool can_reuse(const llm_graph_params & params);
};

class llm_graph_result {
public:
    llm_graph_result(int64_t max_nodes);
    ggml_tensor * get_logits() const;
    ggml_tensor * get_embd() const;
    ggml_cgraph * get_gf() const;
    void set_inputs(const llama_ubatch * ubatch);
    bool can_reuse(const llm_graph_params & params);
    llm_graph_input_i * add_input(llm_graph_input_ptr input);
};

struct llm_graph_context {
    const llm_arch arch;
    const llama_hparams & hparams;
    const llama_cparams & cparams;
    const llama_ubatch & ubatch;
    const int64_t n_embd, n_layer, n_head, n_head_kv;

    ggml_tensor * build_norm(ggml_tensor * cur, ggml_tensor * mw, ggml_tensor * mb,
                              llm_norm_type type, int il);
    ggml_tensor * build_ffn(ggml_tensor * cur, /* weights */,
                             llm_ffn_op_type type_op, llm_ffn_gate_type type_gate);
    ggml_tensor * build_attn(/* attention params */);
    ggml_tensor * build_moe_ffn(/* MoE params */);
    ggml_tensor * build_pooling(ggml_tensor * cur);
};

Import

#include "llama-graph.h"

I/O Contract

Inputs

Name	Type	Required	Description
arch	llm_arch	Yes	Model architecture for graph construction
hparams	llama_hparams	Yes	Model hyperparameters (layers, heads, embedding dims)
ubatch	llama_ubatch	Yes	Micro-batch providing input data
max_nodes	int64_t	Yes	Maximum number of nodes in the compute graph

Outputs

Name	Type	Description
gf	ggml_cgraph *	Computation graph for backend execution
t_logits	ggml_tensor *	Output logits tensor node
t_embd	ggml_tensor *	Output embeddings tensor node

Usage Examples

#include "llama-graph.h"

// In a model's build_graph implementation:
void build_graph(llm_graph_context & ctx) {
    auto * inp = ctx.build_inp_embd(model.tok_embd);

    for (int il = 0; il < n_layer; ++il) {
        // Attention normalization
        auto * cur = ctx.build_norm(inp, model.layers[il].attn_norm, nullptr,
                                     LLM_NORM_RMS, il);
        // Self-attention
        cur = ctx.build_attn(/* params */);
        // FFN
        cur = ctx.build_ffn(cur, /* weights */, LLM_FFN_SILU, LLM_FFN_PAR);
        inp = cur;
    }
    // Final normalization and output
    ctx.build_pooling(inp);
}

Related Pages

• Principle:Ollama_Ollama_Compute_Graph_System