Principle:Ggml org Llama cpp Embedding Model Loading

Field	Value
Principle Name	Embedding Model Loading
Domain	Model Initialization, Embedding Configuration
Description	Theory of loading models specifically configured for embedding extraction: pooling types, embedding flag, and context parameter setup
Related Workflow	Embedding_Extraction

Overview

Description

The Embedding Model Loading principle defines the theoretical foundation for initializing a language model in a configuration suitable for extracting dense vector representations rather than generating text. While the same model weights may support both generation and embedding tasks, the runtime configuration differs significantly. Embedding-mode loading requires enabling the embedding output flag, selecting an appropriate pooling strategy, and configuring context parameters to maximize batch throughput.

Key configuration aspects include:

• Embedding flag: The embeddings boolean in context parameters must be set to true to instruct the model to compute and retain hidden state vectors rather than only logits.
• Pooling type selection: Determines how per-token hidden states are aggregated into a single vector per input sequence. Options include no pooling (per-token embeddings), mean pooling, CLS token pooling, last token pooling, and rank pooling (for reranking models).
• Batch size alignment: For non-causal (embedding) models, the micro-batch size (n_ubatch) must equal the batch size (n_batch) because all tokens in a sequence must be processed together to compute correct pooled embeddings.
• Context size utilization: The batch size should be set to at least the context size to process maximum-length inputs without truncation.

Usage

Embedding model loading is the first step in any embedding extraction workflow. It applies when:

• Setting up the standalone llama-embedding CLI tool
• Configuring llama-server with the --embedding flag
• Programmatically initializing embedding extraction through the C API

Theoretical Basis

Dual-mode model operation is possible because transformer models naturally produce both per-token hidden states and vocabulary logits. The embedding flag controls whether the runtime retains the intermediate hidden states (embeddings) in addition to or instead of the final logits layer output. When embeddings are enabled, the computation graph is modified to store the output of the final transformer layer before the language model head.

Pooling strategies reduce variable-length token sequences to fixed-dimensional vectors:

• LLAMA_POOLING_TYPE_NONE: No pooling; returns per-token embeddings. Useful for token-level tasks like NER or when the caller implements custom pooling.
• LLAMA_POOLING_TYPE_MEAN: Averages all token embeddings in the sequence. The most common strategy for sentence embeddings.
• LLAMA_POOLING_TYPE_CLS: Uses only the embedding of the [CLS] or first token. Standard for BERT-family models.
• LLAMA_POOLING_TYPE_LAST: Uses the embedding of the last token. Common for causal models used as encoders.
• LLAMA_POOLING_TYPE_RANK: Specialized for reranking models; attaches a classification head and returns relevance scores rather than embedding vectors.

Batch size constraints arise from the attention computation in non-causal models. Unlike autoregressive generation where tokens can be processed incrementally, embedding models with bidirectional attention require all tokens in a sequence to attend to each other simultaneously. This means the entire input must fit within a single micro-batch.

Related Pages

• Implementation:Ggml_org_Llama_cpp_Llama_Model_Load_For_Embeddings