Implementation:Mlc ai Mlc llm Serve Data Header

Knowledge Sources	Mlc_ai_Mlc_llm
Domains	LLM Serving, Multi-Modal Data, Data Structures
Last Updated	2026-02-09 19:00 GMT

Overview

The Serve Data Header defines the multi-modal data type hierarchy and streaming output structures used throughout the MLC LLM serving engine. It establishes a polymorphic data abstraction that uniformly handles text, tokens, and images, along with sampling result types and incremental streaming output objects.

Description

This header file (cpp/serve/data.h) defines the following key types in the mlc::llm::serve namespace:

Data type hierarchy (TVM Object-based):

• DataNode (abstract base class): Defines the interface for all data types with two pure virtual methods:
  • GetLength() -- returns the equivalent token count
  • GetEmbedding(model, dst, offset) -- computes or retrieves embeddings, optionally writing them in-place to a destination array at a given offset
• TextDataNode: Holds a text string. Neither GetLength() nor GetEmbedding() is supported directly -- text must be tokenized first.
• TokenDataNode: Holds an IntTuple of token IDs. Length is the number of tokens. Embeddings are computed via model->TokenEmbed().
• ImageDataNode: Holds a Tensor of pixel values and an embed_size. Length equals the embed size. Embeddings are computed via model->ImageEmbed().

Each node class has a corresponding reference class (Data, TextData, TokenData, ImageData) following the TVM object reference pattern.

Utility function:

• SplitData: Splits an Array into two arrays at a specified token position, supporting partial truncation of TokenData.

Sampling types:

• TokenProbPair: A type alias for std::pair<int32_t, float> representing a token ID and its probability.
• SampleResult: A plain struct (not a TVM object) holding the sampled token and a vector of top-probability tokens. Provides methods to get the token ID and generate OpenAI-compatible logprob JSON.

Streaming output:

• RequestStreamOutputObj: A mutable TVM object representing incremental generation output, containing:
  • request_id -- the request identifier
  • group_delta_token_ids -- new tokens per output group
  • group_delta_logprob_json_strs -- optional logprob JSON strings
  • group_finish_reason -- per-group finish reasons
  • request_final_usage_json_str -- final usage statistics
  • group_extra_prefix_string -- extra prefix strings
  • unpacked -- an atomic flag ensuring the object is unpacked at most once
• RequestStreamOutput: The managed reference class with constructors and a Usage static factory method.

Usage

These data types form the input/output contract of the serving pipeline. Requests carry Array as their input content. The engine processes each element through embedding computation, and results stream back through RequestStreamOutput objects carrying delta tokens and logprobs.

Code Reference

Source Location

Property	Value
File	cpp/serve/data.h
Namespace	mlc::llm::serve
Lines	255
Include Guard	MLC_LLM_SERVE_DATA_H_

Signature

namespace mlc {
namespace llm {
namespace serve {

// Abstract base
class DataNode : public Object {
 public:
  virtual int GetLength() const = 0;
  virtual ObjectRef GetEmbedding(Model model, ObjectRef* dst = nullptr, int offset = 0) const = 0;
};
class Data : public ObjectRef { /* ... */ };

// Text data
class TextDataNode : public DataNode {
 public:
  tvm::ffi::String text;
  int GetLength() const final;
  ObjectRef GetEmbedding(Model model, ObjectRef* dst, int offset) const final;
};
class TextData : public Data {
 public:
  explicit TextData(String text);
};

// Token data
class TokenDataNode : public DataNode {
 public:
  IntTuple token_ids;
  int GetLength() const final;
  ObjectRef GetEmbedding(Model model, ObjectRef* dst, int offset) const final;
};
class TokenData : public Data {
 public:
  explicit TokenData(IntTuple token_ids);
  explicit TokenData(std::vector<int32_t> token_ids);
};

// Image data
class ImageDataNode : public DataNode {
 public:
  Tensor image;
  int embed_size;
  int GetLength() const final;
  ObjectRef GetEmbedding(Model model, ObjectRef* dst, int offset) const final;
};
class ImageData : public Data {
 public:
  explicit ImageData(Tensor image, int embed_size);
};

// Split utility
std::pair<Array<Data>, Array<Data>> SplitData(
    const Array<Data>& original_data, int total_length, int split_pos);

// Sampling result
using TokenProbPair = std::pair<int32_t, float>;
struct SampleResult {
  TokenProbPair sampled_token_id;
  std::vector<TokenProbPair> top_prob_tokens;
  int32_t GetTokenId() const;
  std::string GetLogProbJSON(const Tokenizer& tokenizer, bool logprob) const;
};

// Streaming output
class RequestStreamOutputObj : public Object {
 public:
  String request_id;
  std::vector<std::vector<int64_t>> group_delta_token_ids;
  std::optional<std::vector<std::vector<String>>> group_delta_logprob_json_strs;
  std::vector<Optional<String>> group_finish_reason;
  Optional<String> request_final_usage_json_str;
  std::vector<String> group_extra_prefix_string;
  std::atomic<bool> unpacked = false;
};
class RequestStreamOutput : public ObjectRef {
 public:
  explicit RequestStreamOutput(String request_id, /* ... */);
  static RequestStreamOutput Usage(String request_id, String request_final_usage_json_str);
};

}  // namespace serve
}  // namespace llm
}  // namespace mlc

Import

#include "serve/data.h"

Dependencies:

• TVM containers: tvm/ffi/container/array.h, tvm/ffi/container/shape.h
• TVM types: tvm/ffi/optional.h, tvm/ffi/string.h, tvm/runtime/int_tuple.h, tvm/runtime/object.h, tvm/runtime/tensor.h
• tvm/ffi/reflection/registry.h for reflection macros
• tvm/node/cast.h for object casting
• atomic, optional (standard library)
• ../tokenizers/tokenizers.h for the Tokenizer type

I/O Contract

DataNode Interface

Method	Return Type	Description
GetLength()	int	Returns the token-equivalent length of the data
GetEmbedding(model, dst, offset)	ObjectRef	Computes embeddings; optionally writes in-place to dst at offset

Data Type Capabilities

Type	GetLength	GetEmbedding	Notes
TextDataNode	FATAL	FATAL	Must tokenize first
TokenDataNode	Number of token IDs	Via model->TokenEmbed()	Supports partial truncation in SplitData
ImageDataNode	embed_size	Via model->ImageEmbed()	Fixed embedding size

RequestStreamOutputObj Fields

Field	Type	Description
request_id	String	Request identifier
group_delta_token_ids	std::vector<std::vector<int64_t>>	New tokens per parallel output group
group_delta_logprob_json_strs	std::optional<std::vector<std::vector<String>>>	Log probability JSON strings (optional)
group_finish_reason	std::vector<Optional<String>>	Finish reason per group (None if ongoing)
request_final_usage_json_str	Optional<String>	Final usage statistics JSON
group_extra_prefix_string	std::vector<String>	Extra prefix strings per group
unpacked	std::atomic<bool>	Guard ensuring single unpack

Usage Examples

Creating different data types:

#include "serve/data.h"

// Text data (must be tokenized before use)
TextData text_data("Hello, world!");

// Token data from vector
std::vector<int32_t> tokens = {1, 2, 3, 4};
TokenData token_data(tokens);
int len = token_data->GetLength();  // 4

// Image data from tensor
Tensor pixel_values = /* preprocessed image tensor */;
ImageData image_data(pixel_values, 576);  // 576 embedding tokens

Building a streaming output:

RequestStreamOutput output(
    /*request_id=*/"req-001",
    /*group_delta_token_ids=*/{{42, 17}},
    /*group_delta_logprob_json_strs=*/std::nullopt,
    /*group_finish_reason=*/{Optional<String>()},
    /*group_extra_prefix_string=*/{""}
);

// Usage-only output
RequestStreamOutput usage_output = RequestStreamOutput::Usage("req-001", usage_json);

Related Pages

• Mlc_ai_Mlc_llm_Serve_Data - The implementation file for these data types
• Mlc_ai_Mlc_llm_Engine_Interface - The engine that processes data objects
• Mlc_ai_Mlc_llm_Engine_Action - Engine actions that operate on data during prefill and decode
• Mlc_ai_Mlc_llm_OpenAI_API_Protocol_Header - Protocol that consumes streaming output