Implementation:Mlc ai Mlc llm Convert Tiktoken

Overview

Convert Tiktoken is a support module in MLC LLM that converts tiktoken-based tokenizers (used by OpenAI models) into the HuggingFace Tokenizers format. It is located at python/mlc_llm/support/convert_tiktoken.py (170 lines).

This conversion is necessary because MLC LLM's runtime expects tokenizers in the HuggingFace format, while some models (notably those from OpenAI and similar providers) ship with tiktoken tokenizers that use a different BPE encoding scheme.

Purpose

The module adapts the tiktoken BPE encoding into a format compatible with HuggingFace's tokenizers library. It generates five output files that together constitute a complete HuggingFace-compatible tokenizer:

• vocab.json -- The vocabulary mapping tokens to IDs
• tokenizer.json -- The full tokenizer configuration including BPE model, pre/post-processors, and decoder
• tokenizer_config.json -- Metadata about the tokenizer class and settings
• special_tokens_map.json -- Mapping of special token roles (BOS, EOS, UNK)
• merges.txt -- The BPE merge rules

Core Functions

bpe

def bpe(
    mergeable_ranks: Dict[bytes, int],
    token: bytes,
    max_rank: Optional[int] = None,
) -> List[bytes]:

Implements the byte-pair encoding algorithm for a single token. This is adapted from the [community discussion].

The algorithm works by:

1. Splitting the token into individual bytes.
2. Iteratively finding the byte pair with the lowest merge rank.
3. Merging that pair into a single unit.
4. Repeating until no more merges are possible (or the max_rank threshold is reached).

The max_rank parameter is important for generating merge rules: when called with a specific rank limit, the function stops merging before the final merge, revealing the two components that were merged to form the token.

generate_vocab_and_merges

def generate_vocab_and_merges(encoder, mergeable_ranks):

Generates the vocabulary dictionary and merge rules list in HuggingFace format. It uses the GPT-2 bytes_to_unicode mapping from the transformers library to convert raw byte tokens into printable Unicode strings:

from transformers.models.gpt2.tokenization_gpt2 import bytes_to_unicode

byte_encoder = bytes_to_unicode()

def token_bytes_to_string(b):
    return "".join([byte_encoder[ord(char)] for char in b.decode("latin-1")])

For each multi-byte token in the vocabulary, the function calls bpe() with max_rank=rank to determine the two sub-tokens that were merged. These are recorded as merge rules. Special tokens from the encoder are added to the vocabulary separately.

convert_tiktoken

def convert_tiktoken(model_path, output_dir, context_window_size=None):

The main entry point that orchestrates the full conversion process:

1. Loads the tiktoken tokenizer from the model path using AutoTokenizer.from_pretrained with trust_remote_code=True.
2. Extracts the underlying tiktoken encoder and vocabulary.
3. Calls generate_vocab_and_merges to produce HuggingFace-compatible vocabulary and merge rules.
4. Constructs the added_tokens list from special tokens, each marked with special: True.
5. Builds the tokenizer.json template using a ByteLevel BPE model configuration.
6. Builds the tokenizer_config.json with default special tokens set to <|endoftext|>.
7. Writes all five output files to output_dir.

Tokenizer Template Structure

The generated tokenizer.json follows this structure:

tokenizer_template = {
    "version": "1.0",
    "truncation": None,
    "padding": None,
    "added_tokens": added_tokens,
    "normalizer": None,
    "pre_tokenizer": {
        "type": "ByteLevel",
        "add_prefix_space": False,
        "trim_offsets": True,
        "use_regex": True,
    },
    "post_processor": {
        "type": "ByteLevel",
        "add_prefix_space": True,
        "trim_offsets": False,
        "use_regex": True,
    },
    "decoder": {
        "type": "ByteLevel",
        "add_prefix_space": True,
        "trim_offsets": True,
        "use_regex": True,
    },
    "model": {
        "type": "BPE",
        "dropout": None,
        "unk_token": None,
        "continuing_subword_prefix": "",
        "end_of_word_suffix": "",
        "fuse_unk": False,
        "byte_fallback": False,
        "vocab": vocab,
        "merges": merges,
    },
}

Key aspects of the configuration:

• Pre-tokenizer: ByteLevel without prefix space, matching tiktoken's behavior.
• Post-processor: ByteLevel with prefix space added.
• Decoder: ByteLevel to reconstruct original text from tokenized bytes.
• Model: BPE with no dropout, no unknown token, and no byte fallback.

Default Special Tokens

The conversion uses <|endoftext|> as the default for all three special token roles:

{
    "bos_token": "<|endoftext|>",
    "eos_token": "<|endoftext|>",
    "unk_token": "<|endoftext|>",
}

Dependencies

• json, os -- Standard library for file I/O
• transformers.AutoTokenizer -- Used to load the tiktoken tokenizer (conditionally imported)
• transformers.models.gpt2.tokenization_gpt2.bytes_to_unicode -- Byte-to-Unicode mapping for GPT-2 style encoding

File Location

python/mlc_llm/support/convert_tiktoken.py