Implementation:Ggml org Llama cpp Retrieval
| Knowledge Sources | |
|---|---|
| Domains | Embedding, Retrieval |
| Last Updated | 2026-02-15 00:00 GMT |
Overview
Implements a simple text retrieval system using embedding-based cosine similarity to find relevant document chunks matching a query.
Description
Reads multiple context files, splits them into chunks based on a configurable separator and minimum size. Tokenizes and batch-embeds all chunks using an embedding model. Enters an interactive loop where user queries are embedded and compared against all chunk embeddings via cosine similarity. Returns the top-k most similar chunks with their source file, position, similarity score, and text content.
Usage
Use this example as a reference implementation for RAG (Retrieval-Augmented Generation) retrieval pipelines. It demonstrates how to use llama.cpp's embedding capabilities for document search and retrieval applications.
Code Reference
Source Location
- Repository: Ggml_org_Llama_cpp
- File: examples/retrieval/retrieval.cpp
- Lines: 1-304
Signature
struct chunk {
std::string filename;
size_t filepos;
std::string textdata;
std::vector<llama_token> tokens;
std::vector<float> embedding;
};
static std::vector<chunk> chunk_file(const std::string & filename, int chunk_size, const std::string & chunk_separator);
static void batch_add_seq(llama_batch & batch, const std::vector<int32_t> & tokens, llama_seq_id seq_id);
static void batch_decode(llama_context * ctx, llama_batch & batch, float * output, int n_seq, int n_embd, int embd_norm);
int main(int argc, char ** argv);
Import
#include "arg.h"
#include "common.h"
#include "log.h"
#include "llama.h"
#include <algorithm>
#include <fstream>
#include <iostream>
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| params.model | std::string | Yes | Path to an embedding model GGUF file (e.g., bge-base-en) |
| params.context_files | vector<string> | Yes | List of text files to index as the retrieval corpus |
| params.chunk_size | int | No | Minimum size of text chunks for splitting documents |
| params.chunk_separator | std::string | No | Separator string used to split documents into chunks (e.g., ".") |
| params.n_top_k | int | No | Number of top-k most similar chunks to return per query |
| query | stdin | Yes | Interactive user query typed at the prompt |
Outputs
| Name | Type | Description |
|---|---|---|
| results | stdout | Top-k matching chunks with similarity score, source filename, file position, and text content |
Usage Examples
# Index README and LICENSE files, then interactively query
./llama-retrieval \
--model ./models/bge-base-en-v1.5-f16.gguf \
--top-k 3 \
--context-file README.md \
--context-file LICENSE \
--chunk-size 100 \
--chunk-separator .
# Then type queries interactively:
# > What license is this project under?
# Results will show top-3 most similar chunks with scores