Implementation:Ollama Ollama Mtmd

Knowledge Sources	Ollama
Domains	Multimodal, CLIP
Last Updated	2025-02-15 00:00 GMT

Overview

Core implementation of the libmtmd multimodal library that bridges vision/audio encoders with the LLM text pipeline, handling tokenization, encoding, and input chunk management.

Description

Defines internal data structures: mtmd_bitmap for raw images/audio, mtmd_image_tokens / mtmd_audio_tokens for preprocessed media with position metadata, mtmd_input_chunk / mtmd_input_chunks for mixed text-media sequences. Implements mtmd_context which manages separate CLIP encoder contexts for vision and audio, model initialization with warmup, and media marker detection. The mtmd_tokenizer class handles splitting interleaved text and media markers into properly ordered chunks, with model-specific handling for slice templates (MiniCPM-V 2.5/2.6, Llama 4, Idefics3) and special token insertion. Provides the full C API for context lifecycle, bitmap management, tokenization, encoding, and output retrieval.

Usage

Central coordination layer used by Ollama's Go code (via the CGo bridge) to process multimodal inputs including images and audio alongside text prompts.

Code Reference

Source Location

• Repository: Ollama
• File: llama/llama.cpp/tools/mtmd/mtmd.cpp
• Lines: 1-1129

Signature

struct mtmd_bitmap {
    uint32_t nx, ny;
    std::vector<unsigned char> data;
    std::string id;
    bool is_audio = false;
};

struct mtmd_image_tokens {
    uint32_t nx, ny;
    bool use_mrope_pos = false;
    uint32_t n_tokens() const { return nx * ny; }
    clip_image_f32_batch batch_f32;
};

struct mtmd_context {
    struct clip_ctx * ctx_v;  // vision
    struct clip_ctx * ctx_a;  // audio
    const struct llama_model * text_model;
    std::string media_marker;
    mtmd_slice_tmpl slice_tmpl;
};

mtmd_context_params mtmd_context_params_default();
mtmd_context * mtmd_init_from_file(const char * mmproj_fname,
    const struct llama_model * text_model,
    const struct mtmd_context_params ctx_params);
void mtmd_free(mtmd_context * ctx);

Import

#include "mtmd.h"
#include "clip.h"
#include "clip-impl.h"
#include "llama.h"

I/O Contract

Inputs

Name	Type	Required	Description
mmproj_fname	const char *	Yes	Path to the multimodal projector GGUF file
text_model	llama_model *	Yes	Loaded LLM text model for tokenization
text	mtmd_input_text *	Yes	Input text with media markers
bitmaps	mtmd_bitmap **	Yes	Array of image/audio bitmaps matching markers

Outputs

Name	Type	Description
mtmd_context	struct *	Initialized multimodal context
chunks	mtmd_input_chunks *	Tokenized sequence of text and media chunks
embd	float *	Encoded media embeddings for LLM consumption

Usage Examples

// Initialize mtmd context
auto params = mtmd_context_params_default();
mtmd_context * ctx = mtmd_init_from_file("mmproj.gguf", text_model, params);

// Tokenize mixed text+image input
mtmd_input_text * text = mtmd_input_text_init(
    "Describe this image: <__media__>", true, true);
mtmd_input_chunks * chunks = mtmd_input_chunks_init();
mtmd_tokenize(ctx, chunks, text, &bitmap, 1);

// Encode media chunks
mtmd_encode(ctx, mtmd_input_chunks_get(chunks, 1));
float * embd = mtmd_get_output_embd(ctx);

mtmd_free(ctx);

Related Pages

• Principle:Ollama_Ollama_MultimodalPipeline