Implementation:Mlc ai Mlc llm Model Metadata Impl

Knowledge Sources	Mlc_ai_Mlc_llm
Domains	LLM Serving, Model Metadata, JSON Parsing
Last Updated	2026-02-09 19:00 GMT

Overview

The Model Metadata Implementation file provides the JSON deserialization logic for the ModelMetadata struct and its nested types. It reads model metadata from JSON objects (typically embedded in compiled model libraries) and populates the C++ data structures that the rest of the MLC LLM system uses to understand model characteristics such as parameter shapes, quantization, context window size, and KV cache configuration.

Description

This source file (cpp/metadata/model.cc) implements the FromJSON factory methods declared in the corresponding header. It contains five key deserialization functions:

• Param::Preproc::FromJSON: Parses preprocessing metadata for a model parameter, including the preprocessing function name, input/output shapes (which may be symbolic and are resolved against the model config), and output data type.
• Param::FromJSON: Parses a single parameter entry, including its name, data type, shape (resolved from symbolic shapes), preprocessing stages, and pipeline parallel stage assignments.
• KVCacheMetadata::FromJSON: Parses the KV cache configuration containing the number of hidden layers, head dimension, number of attention heads, and number of key-value heads.
• ModelMetadata::FromJSON: The main deserialization method that parses all top-level metadata fields including model type, quantization scheme, context window size, prefill chunk size, max batch size, sliding window parameters, tensor/pipeline parallelism, disaggregation mode, KV state kind, all parameters, and memory usage per function.
• ModelMetadata::FromModule: A convenience method that extracts the JSON metadata string from a compiled TVM module by calling its _metadata function, then delegates to FromJSON.

Usage

Model metadata is typically loaded during engine initialization:

1. The engine loads a compiled model library (TVM Module).
2. ModelMetadata::FromModule is called to extract and parse the embedded metadata.
3. The resulting ModelMetadata struct informs engine configuration decisions such as KV cache allocation, batch size limits, and parallelism strategies.

Code Reference

Source Location

Property	Value
File	cpp/metadata/model.cc
Namespace	mlc::llm
Lines	142
Implements	ModelMetadata declared in cpp/metadata/model.h

Signature

namespace mlc {
namespace llm {

// Preproc deserialization
ModelMetadata::Param::Preproc ModelMetadata::Param::Preproc::FromJSON(
    const picojson::object& js, const picojson::object& model_config);

// Param deserialization
ModelMetadata::Param ModelMetadata::Param::FromJSON(
    const picojson::object& param, const picojson::object& model_config);

// KV cache metadata deserialization
ModelMetadata::KVCacheMetadata ModelMetadata::KVCacheMetadata::FromJSON(
    const picojson::object& json);

// Full model metadata deserialization
ModelMetadata ModelMetadata::FromJSON(
    const picojson::object& metadata, const picojson::object& model_config);

// Load metadata from a compiled TVM module
ModelMetadata ModelMetadata::FromModule(
    Module module, const picojson::object& model_config);

}  // namespace llm
}  // namespace mlc

Import

#include "metadata/model.h"

Dependencies:

• ./model.h (the corresponding header)
• ../support/json_parser.h for json::Lookup, json::LookupOrDefault, json::LookupOptional, and json::SymShapeTuple
• unordered_map (standard library)
• TVM runtime types: tvm::ffi::Function, tvm::ffi::Optional

I/O Contract

ModelMetadata::FromJSON

Direction	Name	Type	Description
Input	metadata	const picojson::object&	JSON object containing model metadata fields
Input	model_config	const picojson::object&	JSON object with model config for resolving symbolic shapes
Output	(return)	ModelMetadata	Fully populated metadata struct

Required JSON fields in metadata:

Field	Type	Description
model_type	string	The type of model (e.g., "llama")
quantization	string	Quantization scheme (e.g., "q4f16_1")
context_window_size	int64	Maximum context length
prefill_chunk_size	int64	Chunk size for prefill
max_batch_size	int64	Maximum batch size supported
tensor_parallel_shards	int64	Number of tensor parallel shards
params	array	Array of parameter metadata objects
memory_usage	object	Map of function name to memory usage in bytes

Optional JSON fields:

Field	Type	Default	Description
sliding_window_size	int64	(none)	Sliding window attention size
attention_sink_size	int64	(none)	Attention sink size
seqlen_padding_factor	int64	1	Sequence length padding factor
pipeline_parallel_stages	int64	1	Number of pipeline parallel stages
disaggregation	bool	false	Whether disaggregation is enabled
kv_state_kind	string	"kv_cache"	Kind of KV state: "kv_cache", "rnn_state", or "none"

ModelMetadata::FromModule

Direction	Name	Type	Description
Input	module	Module	A compiled TVM module that exposes a _metadata function
Input	model_config	const picojson::object&	Model config for shape resolution
Output	(return)	ModelMetadata	Parsed metadata from the module

Usage Examples

Loading metadata from a TVM module:

#include "metadata/model.h"

tvm::ffi::Module model_lib = /* load compiled model library */;
picojson::object model_config = /* parsed model config JSON */;

ModelMetadata metadata = ModelMetadata::FromModule(model_lib, model_config);
// Access metadata.model_type, metadata.context_window_size, etc.
int64_t ctx_size = metadata.context_window_size;
int64_t num_layers = metadata.kv_cache_metadata.num_hidden_layers;

Parsing metadata from a JSON object:

picojson::object metadata_json = /* parsed from JSON string */;
picojson::object model_config = /* model config */;
ModelMetadata metadata = ModelMetadata::FromJSON(metadata_json, model_config);

for (const auto& param : metadata.params) {
  LOG(INFO) << "Parameter: " << param.name << " dtype: " << param.dtype;
}

Related Pages

• Mlc_ai_Mlc_llm_Model_Metadata_Header - The header declaring the ModelMetadata struct
• Mlc_ai_Mlc_llm_Engine_Interface - The engine that uses model metadata during initialization
• Mlc_ai_Mlc_llm_Engine_Action - Engine actions that reference model configuration