Implementation:Mlc ai Mlc llm Tokenizer Header

Overview

The file cpp/tokenizers/tokenizers.h is the C++ header that defines the tokenizer abstraction layer for the MLC LLM runtime. It declares the classes TokenizerInfo, TokenizerObj, and Tokenizer, which together provide the interface for encoding text into token IDs and decoding token IDs back into text. The header wraps the underlying tokenizers_cpp library and integrates with TVM's object system via tvm::runtime::Object and tvm::ffi facilities.

Location

• Repository: Mlc_ai_Mlc_llm
• File: cpp/tokenizers/tokenizers.h
• Lines: 166

Key Components

Include Dependencies

The header depends on:

• tokenizers_cpp.h -- The underlying C++ tokenizer library that provides the core tokenization logic.
• tvm/ffi/container/array.h, tvm/ffi/reflection/registry.h, tvm/ffi/string.h, tvm/runtime/object.h -- TVM FFI and runtime headers for object reference counting, array containers, and reflection.
• ../base.h -- Provides the MLC_LLM_DLL export macro.
• ../support/dynamic_bitset.h -- Provides the DynamicBitset type used for prefix token masks.

TokenizerInfoNode Class

TokenizerInfoNode extends tvm::runtime::Object and holds metadata about how the tokenizer processes tokens:

class TokenizerInfoNode : public Object {
 public:
  String token_postproc_method = "byte_fallback";
  bool prepend_space_in_encode = false;
  bool strip_space_in_decode = false;

  String AsJSONString() const;

  static void RegisterReflection() {
    namespace refl = tvm::ffi::reflection;
    refl::ObjectDef<TokenizerInfoNode>();
  }
  // ...
  TVM_FFI_DECLARE_OBJECT_INFO("mlc.serve.TokenizerInfo", TokenizerInfoNode, Object);
};

The field token_postproc_method controls how raw tokens are decoded back to their original string representation. Two methods are supported:

• "byte_fallback" -- Used by tokenizers such as LLaMA-2 and Mixtral-7B. It transforms byte-fallback tokens (e.g., <0x1B>) to their hex character equivalents and replaces the Unicode character U+2581 ("_") with a space.
• "byte_level" -- Used by tokenizers such as LLaMA-3, GPT-2, and Phi-2. It reverses the bytes-to-unicode mapping defined in the GPT-2 tokenizer.

The boolean fields prepend_space_in_encode and strip_space_in_decode control whether a leading space is added during encoding and whether it is stripped during decoding, respectively.

TokenizerInfo Reference Class

TokenizerInfo is the lightweight managed reference to TokenizerInfoNode:

class TokenizerInfo : public ObjectRef {
 public:
  static TokenizerInfo FromJSONString(String json_string);
  TVM_FFI_DEFINE_OBJECT_REF_METHODS_NULLABLE(TokenizerInfo, ObjectRef, TokenizerInfoNode);
};

The static method FromJSONString constructs a TokenizerInfo object from a serialized JSON string.

TokenizerObj Class

TokenizerObj is the core object that wraps a tokenizers::Tokenizer and exposes all tokenization operations:

class TokenizerObj : public Object {
 public:
  std::unique_ptr<tokenizers::Tokenizer> tokenizer;

  std::vector<int32_t> Encode(const std::string& text) const;
  std::vector<int32_t> EncodeNoPrependSpace(const std::string& text) const;
  std::vector<std::vector<int32_t>> EncodeBatch(const Array<String>& texts) const;
  std::string Decode(const std::vector<int32_t>& token_ids) const;
  const std::vector<std::string>& PostProcessedTokenTable();
  const DynamicBitset& GetPrefixTokenMask();
  size_t GetVocabSize() const;
  std::string IdToToken(int32_t token_id) const;
  int32_t TokenToId(const std::string& token) const;
  // ...
};

Key methods:

Method	Description
Encode	Encodes a single text string into a vector of token IDs. May prepend a space depending on tokenizer configuration.
EncodeNoPrependSpace	Encodes text while guaranteeing no leading space is prepended, regardless of tokenizer configuration.
EncodeBatch	Batch-encodes an array of strings into a vector of token ID vectors.
Decode	Decodes a vector of token IDs back into a text string.
PostProcessedTokenTable	Returns the full token table with post-processing applied (e.g., byte-fallback or byte-level decoding). Results are cached.
GetPrefixTokenMask	Returns a DynamicBitset where each bit indicates whether the corresponding token is a prefix of another token. Results are cached.
GetVocabSize	Returns the vocabulary size including special tokens. As noted in the source, this may be smaller than the vocab_size in config.json.
IdToToken	Converts a token ID to its string representation; returns an empty string if the ID is invalid.
TokenToId	Converts a token string to its ID; returns -1 if the token is not found.

The private members include cached data:

 private:
  TokenizerInfo info_;
  std::vector<std::string> post_processed_token_table_;
  DynamicBitset prefix_token_mask_;

Tokenizer Reference Class

Tokenizer is the managed reference to TokenizerObj and provides static factory methods:

class Tokenizer : public ObjectRef {
 public:
  MLC_LLM_DLL static Tokenizer FromPath(const String& path,
                                        std::optional<TokenizerInfo> info = std::nullopt);
  MLC_LLM_DLL static TokenizerInfo DetectTokenizerInfo(const String& path);
  static std::vector<std::string> PostProcessTokenTable(const std::vector<std::string>& token_table,
                                                        const std::string& token_postproc_method);
  // ...
};

Key static methods:

• FromPath -- Loads a tokenizer from a file system path. If info is not provided, the method auto-detects tokenizer metadata.
• DetectTokenizerInfo -- Probes the tokenizer files at the given path and returns a TokenizerInfo describing the tokenizer type.
• PostProcessTokenTable -- Applies the specified post-processing method to a raw token table, returning a table of original string representations.

Namespace and Integration

All classes reside in the mlc::llm namespace and use TVM's object reference mechanism for memory management. The TVM FFI reflection system is registered for both TokenizerInfoNode (as "mlc.serve.TokenizerInfo") and TokenizerObj (as "mlc.Tokenizer"), enabling these objects to be used across language boundaries through TVM's FFI layer.

Design Notes

• The header uses an include guard (MLC_LLM_TOKENIZER_H_) for protection against multiple inclusion.
• The MLC_LLM_DLL macro on FromPath and DetectTokenizerInfo marks these symbols for shared-library export.
• The _type_mutable = true flag on both node classes indicates these are mutable TVM objects, allowing in-place modification of cached fields such as the token table and prefix mask.