Implementation:Mlc ai Mlc llm Tokenizer

Knowledge Sources	Mlc_ai_Mlc_llm
Domains	Tokenization, LLM Serving, Text Processing
Last Updated	2026-02-09 19:00 GMT

Overview

A tokenizer implementation for MLC LLM that provides encoding, decoding, and token post-processing with support for multiple tokenizer formats including HuggingFace, SentencePiece, ByteLevelBPE, and RWKV.

Description

The tokenizers.cc file implements the tokenizer subsystem for MLC LLM, handling the conversion between text and token IDs. It supports multiple tokenizer backends and includes sophisticated token post-processing logic.

TokenizerInfo stores metadata about a tokenizer that affects encoding and decoding behavior:

token_postproc_method: Either "byte_fallback" or "byte_level", determining how raw tokens are decoded back to text.
prepend_space_in_encode: Whether the tokenizer prepends a space before encoding.
strip_space_in_decode: Whether a leading space should be stripped during decoding.

TokenizerObj wraps the underlying tokenizers::Tokenizer (from the tokenizers_cpp library) and provides:

Encode/Decode: Standard text-to-token and token-to-text conversion.
EncodeNoPrependSpace: Encoding without prepending a space, implemented by adding a padding prefix character and stripping the resulting extra tokens.
EncodeBatch: Batch encoding of multiple texts.
PostProcessedTokenTable: Lazily computes and caches the post-processed version of the entire vocabulary.
GetPrefixTokenMask: Computes a bitmask indicating which tokens are prefixes of other tokens, used for prefix-based operations.
GetVocabSize/IdToToken/TokenToId: Vocabulary introspection.

Tokenizer::FromPath is the factory method that auto-detects the tokenizer format from the filesystem. It checks for files in this priority order:

tokenizer.json (HuggingFace format, preferred)
tokenizer.model (SentencePiece format, with a warning)
merges.txt + vocab.json + added_tokens.json (ByteLevelBPE format)
tokenizer_model (RWKV World format)

Tokenizer::DetectTokenizerInfo parses tokenizer.json to auto-detect tokenizer properties by examining the "decoder" and "normalizer" fields. It handles both simple and "Sequence" type decoders/normalizers.

The file also implements three token post-processing decoders:

ByteFallbackDecoder: Converts tokens like "<0x1B>" to their corresponding byte values.
SpaceReplacerDecoder: Replaces the Unicode lower one-eighth block character (U+2581) with a space.
ByteLevelDecoder: Inverts the GPT-2 style bytes-to-unicode mapping using a precomputed 324-entry lookup table.

All functionality is registered with the TVM FFI system, exposing encode, decode, batch encode, and token post-processing operations.

Usage

The tokenizer is a fundamental component used throughout the MLC LLM pipeline. It is used by the serving engine for encoding user inputs, decoding generated tokens for streaming output, and by the grammar-guided generation system for token vocabulary analysis. The auto-detection mechanism allows the same code to work with models from various providers that use different tokenizer formats.

Code Reference

Source Location

Repository: Mlc_ai_Mlc_llm
File: cpp/tokenizers/tokenizers.cc

Signature

class TokenizerObj {
 public:
  std::vector<int32_t> Encode(const std::string& text) const;
  std::vector<int32_t> EncodeNoPrependSpace(const std::string& text) const;
  std::vector<std::vector<int32_t>> EncodeBatch(const Array<String>& texts) const;
  std::string Decode(const std::vector<int32_t>& token_ids) const;
  const DynamicBitset& GetPrefixTokenMask();
  size_t GetVocabSize() const;
  std::string IdToToken(int32_t token_id) const;
  int32_t TokenToId(const std::string& token) const;
  const std::vector<std::string>& PostProcessedTokenTable();
};

class Tokenizer : public ObjectRef {
 public:
  static Tokenizer FromPath(const String& path, std::optional<TokenizerInfo> info = std::nullopt);
  static TokenizerInfo DetectTokenizerInfo(const String& path);
  static std::vector<std::string> PostProcessTokenTable(
      const std::vector<std::string>& token_table, const std::string& token_postproc_method);
};

Import

#include "tokenizers.h"

I/O Contract

Inputs

Name	Type	Required	Description
path	String	Yes (for FromPath)	Path to the tokenizer directory or file.
text	string	Yes (for Encode)	Text string to tokenize.
token_ids	vector<int32_t>	Yes (for Decode)	Token IDs to decode back to text.
info	optional<TokenizerInfo>	No	Optional tokenizer metadata; auto-detected if not provided.
token_postproc_method	string	For PostProcessTokenTable	Either "byte_fallback" or "byte_level".

Outputs

Name	Type	Description
Encode result	vector<int32_t>	Token IDs corresponding to the input text.
Decode result	string	Text string reconstructed from token IDs.
EncodeBatch result	vector<vector<int32_t>>	Batch of token ID sequences.
GetVocabSize result	size_t	Total vocabulary size.
PostProcessedTokenTable result	vector<string>	The entire vocabulary with post-processing applied.
GetPrefixTokenMask result	DynamicBitset	Bitmask where set bits indicate tokens that are prefixes of other tokens.

Usage Examples

// Load a tokenizer from a model directory
Tokenizer tokenizer = Tokenizer::FromPath("/path/to/model");

// Encode text to token IDs
std::vector<int32_t> tokens = tokenizer->Encode("Hello, world!");

// Decode token IDs back to text
std::string text = tokenizer->Decode(tokens);

// Get vocabulary size
size_t vocab_size = tokenizer->GetVocabSize();

// Get post-processed token table for grammar-guided generation
const auto& token_table = tokenizer->PostProcessedTokenTable();

// Batch encode multiple texts
Array<String> texts = {"Hello", "World"};
auto batch_tokens = tokenizer->EncodeBatch(texts);

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment