Implementation:Ollama Ollama Tokenizer BPE
| Knowledge Sources | |
|---|---|
| Domains | Tokenization, Text Processing |
| Last Updated | 2025-02-15 00:00 GMT |
Overview
Implements Byte Pair Encoding (BPE) tokenization, the algorithm used by GPT-style models for text-to-token conversion with configurable pretokenizer patterns.
Description
BytePairEncoding wraps a Vocabulary and a list of compiled regular expressions for pretokenization. Encode first splits input by special tokens, then applies the pretokenizer regexps to split into words. Each word undergoes byte-level encoding (mapping bytes to Unicode codepoints) followed by iterative BPE merging using a priority queue: pairs of tokens are merged by rank (either merge-table rank or vocabulary ID) until no more merges are possible. The BPE algorithm uses a min-heap for efficient merge selection and a linked-list structure for fast token updates during merging.
Usage
Used for models that specify BPE tokenization (GPT, Llama 3, etc.). The default pretokenizer pattern matches English contractions, words, numbers, and whitespace.
Code Reference
Source Location
- Repository: Ollama
- File: tokenizer/bytepairencoding.go
- Lines: 1-282
Signature
type BytePairEncoding struct {
vocab *Vocabulary
regexps []*regexp2.Regexp
}
func NewBytePairEncoding(vocab *Vocabulary, pretokenizer ...string) BytePairEncoding
func (bpe BytePairEncoding) Vocabulary() *Vocabulary
func (bpe BytePairEncoding) Is(id int32, special Special) bool
func (bpe BytePairEncoding) Encode(s string, addSpecial bool) ([]int32, error)
var _ Tokenizer = (*BytePairEncoding)(nil)
Import
import "github.com/ollama/ollama/tokenizer"
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| s | string | Yes | Text to tokenize |
| addSpecial | bool | Yes | Whether to add BOS/EOS tokens |
Outputs
| Name | Type | Description |
|---|---|---|
| ids | []int32 | Token IDs |
| error | error | Encoding error |
Usage Examples
vocab := &tokenizer.Vocabulary{...}
bpe := tokenizer.NewBytePairEncoding(vocab)
ids, err := bpe.Encode("Hello, world!", true)
// ids: [1, 15043, 29892, 3186, 29991, 2] (with BOS/EOS)
// Check if a token is EOS
if bpe.Is(ids[len(ids)-1], tokenizer.SpecialEOS) {
// End of sequence
}