Implementation:Ollama Ollama Llama Vocab

Knowledge Sources	Ollama
Domains	LLM Inference, Tokenization
Last Updated	2025-02-15 00:00 GMT

Overview

Implements the complete vocabulary and tokenization system, supporting multiple tokenizer types (SPM, BPE, WordPiece, Unigram, RWKV, PLaMo2) with their corresponding encoding and decoding algorithms.

Description

Loads vocabulary data from GGUF metadata including token texts, scores, types, special token IDs, and merges. Implements tokenizer-specific algorithms: llm_tokenizer_spm uses SentencePiece unigram model with bigram merging, llm_tokenizer_bpe implements byte-pair encoding with dozens of model-specific regex patterns, llm_tokenizer_wpm provides WordPiece tokenization, llm_tokenizer_ugm implements Unigram language model tokenization with Viterbi-based optimization, llm_tokenizer_rwkv uses trie-based greedy matching, and llm_tokenizer_plamo2 handles PLaMo2-specific tokenization. Uses a naive_trie for efficient prefix matching.

Usage

Essential for all text processing in llama.cpp. Every prompt must be tokenized before inference and every generated token must be detokenized for output. Correct tokenization is critical for model performance since models are trained with specific tokenizers.

Code Reference

Source Location

• Repository: Ollama
• File: llama/llama.cpp/src/llama-vocab.cpp
• Lines: 1-3822

Signature

struct naive_trie {
    void insert(const char * key, size_t len, int32_t value = 0);
    std::pair<const char *, size_t> get_longest_prefix(const char * key, size_t len, size_t offset = 0) const;
    const struct naive_trie * traverse(const char c) const;
};

struct llm_tokenizer { virtual ~llm_tokenizer() = default; };

struct llm_tokenizer_spm_session {
    void tokenize(const std::string & text, std::vector<llama_token> & output);
};

struct llm_tokenizer_bpe_session {
    void tokenize(const std::string & text, std::vector<llama_token> & output);
};

// More tokenizer session types: wpm, ugm, rwkv, plamo2...

Import

#include "llama-vocab.h"

I/O Contract

Inputs

Name	Type	Required	Description
text	const char *	Yes	Input text to tokenize
text_len	int32_t	Yes	Length of input text
add_special	bool	Yes	Whether to add BOS/EOS tokens
parse_special	bool	No	Whether to parse special token strings

Outputs

Name	Type	Description
tokens	llama_token *	Array of output token IDs
n_tokens	int32_t	Number of tokens produced

Usage Examples

// Tokenize text:
auto tokens = vocab.tokenize("Hello, world!", true, false);

// Detokenize:
std::string text = vocab.detokenize(tokens, false);

// Token info:
const char * piece = vocab.token_get_text(token_id);
bool is_special = vocab.is_control(token_id);

Related Pages

• Principle:Ollama_Ollama_Tokenization