Implementation:Mlc ai Mlc llm Encoding

Overview

The Encoding implementation at cpp/support/encoding.cc provides the core logic for encoding and decoding between Unicode codepoints, UTF-8 byte sequences, and C-style escape sequences. It is the implementation counterpart to the declarations in encoding.h and is used throughout MLC LLM wherever text must be converted between these representations, particularly in grammar-guided generation and tokenizer support.

Purpose

This file implements the following capabilities:

• Codepoint to UTF-8 conversion -- Encoding a Unicode codepoint into its multi-byte UTF-8 representation.
• Codepoint to escaped string conversion -- Producing printable representations of codepoints using C-style escape sequences.
• UTF-8 parsing -- Decoding UTF-8 byte sequences into codepoints, with configurable error handling policies.
• Escape sequence parsing -- Interpreting C-style escape sequences (such as \n, \uXXXX) and converting them back to codepoints.

Key Functions

PrintAsUTF8

std::string PrintAsUTF8(TCodepoint codepoint);

Converts a Unicode codepoint to its UTF-8 string encoding. The function handles all four UTF-8 byte lengths:

Codepoint Range	Bytes	Encoding Pattern
U+0000 to U+007F	1	0xxxxxxx
U+0080 to U+07FF	2	110xxxxx 10xxxxxx
U+0800 to U+FFFF	3	1110xxxx 10xxxxxx 10xxxxxx
U+10000 to U+10FFFF	4	11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

An assertion ensures the codepoint does not exceed 0x10FFFF.

PrintAsEscaped

Three overloads are provided:

std::string PrintAsEscaped(TCodepoint codepoint,
    const std::unordered_map<TCodepoint, std::string>& additional_escape_map = {});
std::string PrintAsEscaped(uint8_t raw_char);
std::string PrintAsEscaped(std::string raw_str);

The codepoint variant first checks an optional user-supplied escape map, then a built-in map of standard C escape sequences (including \n, \t, \0, \e, etc.). Printable ASCII characters (0x20 through 0x7E) are returned as-is. All other codepoints are rendered as hex escapes using the appropriate prefix: \xHH for values up to 0xFF, \uHHHH for values up to 0xFFFF, and \UHHHHHHHH for larger values.

The string variant parses the input as UTF-8 first using ParseUTF8, then escapes each resulting codepoint individually.

HandleUTF8FirstByte

std::tuple<bool, int, TCodepoint> HandleUTF8FirstByte(uint8_t byte);

Analyzes the first byte of a UTF-8 sequence. Returns a tuple of:

• Whether the byte is a valid UTF-8 leading byte.
• The total number of bytes in the sequence (1 through 4).
• The initial codepoint value extracted from the first byte using the appropriate mask.

This function uses a 256-entry lookup table (kUtf8Bytes) that maps each possible byte value to the expected UTF-8 sequence length, with -1 indicating an invalid leading byte (continuation bytes 0x80-0xBF and illegal bytes 0xF8-0xFF).

ParseNextUTF8

std::pair<TCodepoint, const char*> ParseNextUTF8(const char* utf8, UTF8ErrorPolicy error_policy);

Parses the next codepoint from a UTF-8 string. After handling the first byte via HandleUTF8FirstByte, it reads the expected number of continuation bytes, validating that each has the form 10xxxxxx. Error handling is controlled by the UTF8ErrorPolicy:

• kReturnInvalid -- Returns CharHandlingError::kInvalidUTF8 and the original pointer.
• kReturnByte -- Returns the raw byte value as a codepoint and advances by one byte.

ParseUTF8

std::vector<TCodepoint> ParseUTF8(const char* utf8, UTF8ErrorPolicy error_policy);

Iterates through an entire null-terminated UTF-8 string, calling ParseNextUTF8 repeatedly to produce a vector of codepoints. If kReturnInvalid is the error policy and an error occurs, the function returns a single-element vector containing the error sentinel.

HexCharToInt

inline int HexCharToInt(char c);

A helper that converts a hexadecimal character (0-9, a-f, A-F) to its integer value (0-15). Returns -1 for invalid characters.

ParseNextUTF8OrEscaped

std::pair<TCodepoint, const char*> ParseNextUTF8OrEscaped(
    const char* utf8, const std::unordered_map<std::string, TCodepoint>& additional_escape_map);

Parses the next character from a string that may contain either raw UTF-8 or escape sequences. If the current character is not a backslash, it delegates to ParseNextUTF8. Otherwise, it interprets the escape sequence in this order:

1. Check the user-supplied additional_escape_map.
2. Check the built-in escape map (\n, \t, \\, etc.).
3. Handle hex escapes: \x (arbitrary-length hex), \u (4-digit hex), and \U (8-digit hex).
4. Return CharHandlingError::kInvalidEscape if none match.

Dependencies

• encoding.h -- Corresponding header with type definitions and function declarations.
• tvm/runtime/logging.h -- Provides ICHECK assertion macros.
• <array> -- Used for the UTF-8 byte lookup table.

File Location

• Source file: cpp/support/encoding.cc
• Namespace: mlc::llm