Implementation:Ollama Ollama Llama Server

Knowledge Sources	Ollama
Domains	Inference Runtime, CGo Bridge
Last Updated	2025-02-15 00:00 GMT

Overview

Provides the primary Go-to-C bridge (via CGo) that exposes the full llama.cpp API as Go types and functions for Ollama's runtime to use for model loading, inference, and sampling.

Description

The llama package wraps llama.cpp's C API using CGo, providing idiomatic Go types for all core operations: backend initialization (BackendInit), GPU device enumeration (EnumerateGPUs), model architecture detection from GGUF files (GetModelArch), model loading with configurable parameters (ModelParams, Model), inference context management (Context, ContextParams), token batch processing (Batch, Decode), sampling with configurable parameters (SamplingContext, SamplingParams), grammar-constrained generation (Grammar), and multimodal support (MtmdContext, MtmdChunk). It also sets up log forwarding from llama.cpp's C logging to Go's slog system.

Usage

Used by the GGML backend implementation as the low-level interface to the llama.cpp inference engine. All model loading, inference execution, sampling, and hardware interaction flows through this package.

Code Reference

Source Location

• Repository: Ollama
• File: llama/llama.go
• Lines: 1-794

Signature

type Model struct { c *C.struct_llama_model }
type Context struct { c *C.struct_llama_context; m *Model }
type Batch struct { c C.struct_llama_batch }
type SamplingContext struct { c *C.struct_llama_sampler }
type Grammar struct { c *C.struct_llama_sampler }
type MtmdContext struct { c *C.struct_mtmd_context }

func BackendInit()
func EnumerateGPUs() Devices
func GetModelArch(modelPath string) (string, error)
func NewContextParams(maxTokens int, ...) ContextParams
func (m *Model) NewContext(params ContextParams) (*Context, error)
func (b *Batch) Add(token int, pos int, seqIds []int, logits bool)
func (c *Context) Decode(batch *Batch) error
func NewSamplingContext(params SamplingParams) *SamplingContext

Import

import "github.com/ollama/ollama/llama"

I/O Contract

Inputs

Name	Type	Required	Description
modelPath	string	Yes	Path to GGUF model file for loading or architecture detection
params	ContextParams	Yes	Configuration for inference context (max tokens, threads, flash attention)
batch	*Batch	Yes	Token batch for decoding with positions and sequence IDs
samplingParams	SamplingParams	Yes	Configuration for token sampling (temperature, top-k, top-p, etc.)

Outputs

Name	Type	Description
*Model	pointer	Loaded llama.cpp model handle
*Context	pointer	Inference context for running forward passes
Devices	struct	GPU device information (count, VRAM, names)
token	int	Sampled token ID from the model's logits

Usage Examples

// Initialize backend and enumerate GPUs
llama.BackendInit()
devices := llama.EnumerateGPUs()

// Load a model
model, err := llama.LoadModel("/path/to/model.gguf", llama.ModelParams{
    NumGPULayers: 35,
})

// Create context and run inference
ctx, err := model.NewContext(llama.NewContextParams(2048, 8, true))
batch := llama.NewBatch(512, 0, 1)
batch.Add(tokenID, position, []int{0}, true)
err = ctx.Decode(&batch)

Related Pages

• Principle:Ollama_Ollama_LlamaCpp_CGo_Bridge