Implementation:Mlc ai Mlc llm ConvTemplate Header

Knowledge Sources	Mlc_ai_Mlc_llm
Domains	C++, LLM, Prompt Engineering, Conversation Management
Last Updated	2026-02-09 19:00 GMT

Overview

conv_template.h declares the conversation template system for MLC LLM's JSON FFI layer, defining data structures for model configuration, vision configuration, message placeholders, and the conversation template used to format prompts for various LLM architectures.

Description

This header file defines the core types and functions within the mlc::llm::json_ffi namespace that govern how chat messages are transformed into model-consumable prompts.

ModelVisionConfig holds the configuration parameters for a model's vision encoder (if present), including hidden_size, image_size, intermediate_size, num_attention_heads, num_hidden_layers, patch_size, projection_dim, vocab_size, dtype, num_channels, and layer_norm_eps. It provides a static FromJSON factory method for deserialization from picojson.

ModelConfig captures the model's structural parameters: vocab_size, context_window_size, sliding_window_size, prefill_chunk_size, tensor_parallel_shards, pipeline_parallel_stages, max_batch_size, and an optional vision_config. It is populated from the "model_config" field in mlc-chat-config.json and also provides a FromJSON factory.

MessagePlaceholders is an enum class with values SYSTEM, USER, ASSISTANT, TOOL, and FUNCTION, used to identify placeholder tokens in templates. The free function MessagePlaceholderFromString converts a role string to the corresponding enum value.

Conversation is the central struct that specifies the conversation template format. Its fields include:

• name -- Optional template name
• system_template -- Template string for the system prompt, with optional placeholder
• system_message -- The actual system prompt content
• system_prefix_token_ids -- Optional token IDs prepended before the tokenized prompt
• add_role_after_system_message -- Controls whether user role and separator follow the system message (for [INST]-style formats)
• roles -- Maps role names to their display strings
• role_templates -- Per-role prompt templates with message placeholders
• messages -- The conversation history as a vector of ChatCompletionMessage
• seps -- Separators between messages (1 or 2 separators supported)
• role_content_sep / role_empty_sep -- Separators between role labels and content
• stop_str / stop_token_ids -- Stop criteria for generation

Key methods:

• GetSystemText -- Applies the system template to a system message
• GetRoleText -- Formats a role's content using the role template, with optional function call string
• FromJSON -- Factory methods to parse from a picojson object or a raw JSON string, returning Result<Conversation>

CreatePrompt is a free function that takes a Conversation, a ChatCompletionRequest, a ModelConfig, and a DLDevice, and produces a Result<std::vector> -- the list of prompt data segments ready for the model.

Usage

This header is included by the JSON FFI engine implementation and any code that needs to format prompts according to a model's conversation template. It supports a wide range of chat model prompt formats (e.g., Llama, ChatML, Vicuna, etc.) through its configurable template system.

Code Reference

Source Location

• Repository: Mlc_ai_Mlc_llm
• File: cpp/json_ffi/conv_template.h

Signature

namespace mlc {
namespace llm {
namespace json_ffi {

class ModelVisionConfig {
 public:
  static ModelVisionConfig FromJSON(const picojson::object& json_obj);
};

class ModelConfig {
 public:
  int vocab_size;
  int context_window_size;
  int sliding_window_size;
  int prefill_chunk_size;
  int tensor_parallel_shards;
  int pipeline_parallel_stages;
  int max_batch_size;
  std::optional<ModelVisionConfig> vision_config;
  static ModelConfig FromJSON(const picojson::object& json_obj);
};

enum class MessagePlaceholders { SYSTEM, USER, ASSISTANT, TOOL, FUNCTION };
MessagePlaceholders MessagePlaceholderFromString(const std::string& role);

struct Conversation {
  std::optional<std::string> name;
  std::string system_template;
  std::string system_message;
  std::optional<std::vector<int>> system_prefix_token_ids;
  bool add_role_after_system_message;
  std::unordered_map<std::string, std::string> roles;
  std::unordered_map<std::string, std::string> role_templates;
  std::vector<ChatCompletionMessage> messages;
  std::vector<std::string> seps;
  std::string role_content_sep;
  std::string role_empty_sep;
  std::vector<std::string> stop_str;
  std::vector<int> stop_token_ids;

  std::string GetSystemText(const std::string& system_msg) const;
  std::string GetRoleText(const std::string& role, const std::string& content,
                          const std::optional<std::string>& fn_call_str) const;
  static Result<Conversation> FromJSON(const picojson::object& json);
  static Result<Conversation> FromJSON(const std::string& json_str);
};

Result<std::vector<Data>> CreatePrompt(const Conversation& conv,
                                       const ChatCompletionRequest& request,
                                       const ModelConfig& config, DLDevice device);

}  // namespace json_ffi
}  // namespace llm
}  // namespace mlc

Import

#include "conv_template.h"

I/O Contract

Function / Method	Input	Output	Description
ModelVisionConfig::FromJSON	picojson::object	ModelVisionConfig	Deserializes vision config from JSON
ModelConfig::FromJSON	picojson::object	ModelConfig	Deserializes model config from JSON
MessagePlaceholderFromString	std::string	MessagePlaceholders	Converts a role string to enum
Conversation::GetSystemText	std::string (system msg)	std::string	Applies system template to message
Conversation::GetRoleText	role, content, optional fn_call	std::string	Formats role content per template
Conversation::FromJSON	JSON object or string	Result<Conversation>	Parses a conversation template from JSON
CreatePrompt	Conversation, Request, Config, Device	Result<std::vector>	Constructs the final prompt data segments

Usage Examples

#include "conv_template.h"

// Parse a conversation template from JSON config
auto conv_result = Conversation::FromJSON(json_config_str);
if (conv_result.IsOk()) {
    Conversation conv = conv_result.Unwrap();

    // Get formatted system text
    std::string sys_text = conv.GetSystemText("You are a helpful assistant.");

    // Get formatted role text
    std::string user_text = conv.GetRoleText("user", "Hello!", std::nullopt);

    // Create prompt data for the model
    auto prompt_result = CreatePrompt(conv, request, model_config, device);
}