Implementation:Mlc ai Mlc llm Model Metadata Header

Knowledge Sources	Mlc_ai_Mlc_llm
Domains	LLM Serving, Model Metadata, Data Structures
Last Updated	2026-02-09 19:00 GMT

Overview

The Model Metadata Header declares the ModelMetadata struct and associated types that describe the static properties of a compiled MLC LLM model library. These properties include model architecture information, quantization details, parallelism configuration, KV cache structure, and per-parameter metadata including shapes and preprocessing steps.

Description

This header file (cpp/metadata/model.h) defines the following types in the mlc::llm namespace:

• KVStateKind enum: Distinguishes between three modes of key-value state management:
  • kKVCache (0) -- standard transformer KV cache
  • kRNNState (1) -- recurrent neural network state
  • kNone (2) -- no KV state (e.g., encoder-only models)
• KVStateKindToString / KVStateKindFromString: Inline utility functions for converting between the enum and its string representation.
• ModelMetadata::Param::Preproc: Describes a preprocessing step for a model parameter, holding the function name, input shape, output shape, and output data type.
• ModelMetadata::Param: Describes a single model parameter with its name, shape, data type, list of preprocessing steps, and pipeline stage assignments.
• ModelMetadata::KVCacheMetadata: Contains the structural properties of the KV cache -- number of hidden layers, head dimension, number of attention heads, and number of key-value heads.
• ModelMetadata: The top-level struct aggregating all model metadata: model type, quantization, context window size, prefill chunk size, max batch size, sliding window size, tensor/pipeline parallelism, disaggregation support, KV state kind, KV cache metadata, all parameters, and per-function memory usage.

Usage

The ModelMetadata struct is a central data structure used throughout the MLC LLM engine initialization process. It is populated either by parsing JSON metadata embedded in a compiled model module (FromModule) or by directly parsing a JSON object (FromJSON). Once populated, it is consulted to:

• Configure the KV cache dimensions and allocation
• Determine batch size and context window constraints
• Set up tensor and pipeline parallelism
• Map parameter names to their shapes and data types for weight loading

Code Reference

Source Location

Property	Value
File	cpp/metadata/model.h
Namespace	mlc::llm
Lines	105
Include Guard	MLC_LLM_CPP_MODEL_METADATA_H_

Signature

namespace mlc {
namespace llm {

using tvm::ffi::Module;
using tvm::ffi::Shape;
using tvm::ffi::String;
using tvm::runtime::DataType;

enum class KVStateKind : int {
  kKVCache = 0,
  kRNNState = 1,
  kNone = 2,
};

inline std::string KVStateKindToString(KVStateKind kv_state_kind);
inline KVStateKind KVStateKindFromString(const std::string& kv_state_kind);

struct ModelMetadata {
  struct Param {
    struct Preproc {
      String func_name;
      Shape in_shape;
      Shape out_shape;
      DataType out_dtype;
      static Preproc FromJSON(const picojson::object& js, const picojson::object& model_config);
    };

    String name;
    Shape shape;
    DataType dtype;
    std::vector<Preproc> preprocs;
    std::vector<int> pipeline_stages;
    static Param FromJSON(const picojson::object& param_obj, const picojson::object& model_config);
  };

  struct KVCacheMetadata {
    int64_t num_hidden_layers;
    int64_t num_attention_heads;
    int64_t num_key_value_heads;
    int64_t head_dim;
    static KVCacheMetadata FromJSON(const picojson::object& json);
  };

  std::string model_type;
  std::string quantization;
  int64_t context_window_size;
  int64_t prefill_chunk_size;
  int64_t max_batch_size;
  int64_t sliding_window_size;
  int64_t tensor_parallel_shards;
  int64_t pipeline_parallel_stages;
  bool disaggregation;
  int64_t attention_sink_size;
  int64_t seqlen_padding_factor;
  std::vector<Param> params;
  std::unordered_map<std::string, int64_t> memory_usage;
  KVStateKind kv_state_kind;
  KVCacheMetadata kv_cache_metadata;

  static ModelMetadata FromJSON(const picojson::object& json_str,
                                const picojson::object& model_config);
  static ModelMetadata FromModule(Module module, const picojson::object& model_config);
};

}  // namespace llm
}  // namespace mlc

Import

#include "metadata/model.h"

Dependencies:

• picojson.h for JSON types
• tvm/ffi/container/shape.h for Shape
• tvm/ffi/extra/module.h for Module
• tvm/ffi/string.h for String
• tvm/runtime/data_type.h for DataType
• tvm/runtime/module.h for TVM runtime module support
• unordered_map (standard library)

I/O Contract

KVStateKind Conversion

Function	Input	Output	Description
KVStateKindToString	KVStateKind enum value	std::string ("kv_cache", "rnn_state", or "none")	Converts enum to string; fatal error on invalid input
KVStateKindFromString	const std::string&	KVStateKind enum value	Converts string to enum; fatal error on unrecognized string

ModelMetadata Fields

Field	Type	Description
model_type	std::string	Architecture name (e.g., "llama", "gpt_neox")
quantization	std::string	Quantization scheme identifier
context_window_size	int64_t	Maximum number of tokens in context
prefill_chunk_size	int64_t	Number of tokens processed per prefill chunk
max_batch_size	int64_t	Maximum concurrent batch size
sliding_window_size	int64_t	Sliding window attention span
tensor_parallel_shards	int64_t	Number of tensor parallel shards
pipeline_parallel_stages	int64_t	Number of pipeline parallel stages
disaggregation	bool	Whether disaggregated serving is supported
attention_sink_size	int64_t	Size of attention sink tokens
seqlen_padding_factor	int64_t	Factor for sequence length padding
params	std::vector<Param>	List of all model parameters
memory_usage	std::unordered_map<std::string, int64_t>	Per-function memory usage in bytes
kv_state_kind	KVStateKind	Type of KV state management
kv_cache_metadata	KVCacheMetadata	Structural KV cache properties

Usage Examples

Accessing model metadata after loading:

#include "metadata/model.h"

ModelMetadata metadata = ModelMetadata::FromModule(model_lib, model_config);

// Check model type and quantization
LOG(INFO) << "Model: " << metadata.model_type
          << " Quantization: " << metadata.quantization;

// Inspect KV cache structure
if (metadata.kv_state_kind == KVStateKind::kKVCache) {
  auto& kv = metadata.kv_cache_metadata;
  LOG(INFO) << "Layers: " << kv.num_hidden_layers
            << " Heads: " << kv.num_attention_heads
            << " KV Heads: " << kv.num_key_value_heads
            << " Head dim: " << kv.head_dim;
}

// Iterate over parameters
for (const auto& param : metadata.params) {
  LOG(INFO) << "Param: " << param.name << " Shape: " << param.shape;
}

Related Pages

• Mlc_ai_Mlc_llm_Model_Metadata_Impl - The implementation file with JSON parsing logic
• Mlc_ai_Mlc_llm_Engine_Interface - The engine interface that uses model metadata
• Mlc_ai_Mlc_llm_Draft_Token_Workspace - Workspace manager that depends on model properties