Implementation:Ggml org Ggml Gpt tokenize
NLP Tokenization GGML BPE Last updated: 2025-05-15 12:00 GMT
Summary
gpt_tokenize is the BPE tokenization function in the GGML library. It takes a vocabulary and an input string, and returns a vector of token IDs by applying the byte-level BPE merging algorithm.
API Signature
std::vector<gpt_vocab::id> gpt_tokenize(const gpt_vocab & vocab, const std::string & text)
Source Location
- File:
examples/common.cpp, lines 259-316 - Repository: https://github.com/ggml-org/ggml
Parameters
| Parameter | Type | Description |
|---|---|---|
vocab |
const gpt_vocab & |
The GPT vocabulary struct containing token_to_id and id_to_token maps
|
text |
const std::string & |
The input text to tokenize |
Return Value
Returns a std::vector<gpt_vocab::id> containing the integer token IDs corresponding to the BPE-encoded subword units of the input text.
Implementation Details
The function operates in two stages:
- Word splitting: The input text is split into individual words using a regex pattern. This isolates whitespace-delimited tokens and punctuation for independent BPE processing.
- BPE merging: For each word, the algorithm iteratively finds the most frequent adjacent symbol pair (according to the vocabulary's merge priority) and merges them, continuing until no more merges are possible. The resulting subword strings are looked up in
vocab.token_to_idto produce integer IDs.
The gpt_vocab Struct
Defined in examples/common.h, lines 62-66:
struct gpt_vocab {
using id = int32_t;
using token = std::string;
std::map<token, id> token_to_id;
std::map<id, token> id_to_token;
};
The struct provides bidirectional mapping between token strings and their integer IDs, supporting both encoding and decoding operations.
Decoding
The reverse operation (token IDs back to text) uses the id_to_token map:
vocab.id_to_token[id].c_str()
This is used in examples/gpt-2/main-backend.cpp, lines 916-918, during text generation to convert predicted token IDs back into readable strings.
Dependencies
- Language: C++
- Standard library:
regex,string,map,vector - No external dependencies beyond the C++ standard library.