Implementation:Mlc ai Mlc llm OpenAI API Protocol Header

Knowledge Sources	Mlc_ai_Mlc_llm
Domains	LLM Serving, API Protocol, JSON FFI
Last Updated	2026-02-09 19:00 GMT

Overview

The OpenAI API Protocol Header defines the C++ data structures that mirror the OpenAI Chat Completion API specification within the MLC LLM framework. This header is part of the JSON FFI (Foreign Function Interface) layer and provides classes for serializing and deserializing chat completion requests and responses using the picojson library.

Description

This header file (cpp/json_ffi/openai_api_protocol.h) establishes the protocol layer that allows MLC LLM to accept and respond to requests conforming to the OpenAI Chat Completion API format. It defines:

• Enumerations: Type (text, json_object, function) and FinishReason (stop, length, tool_calls, error) that represent API-level type and termination semantics.
• ChatFunction / ChatTool / ChatFunctionCall / ChatToolCall: Classes modeling the tool/function calling protocol, including function definitions, tool wrappers, and invocation structures.
• ChatCompletionMessageContent: A variant-like class that can hold either plain text, a null value, or multi-part content (a vector of string key-value maps).
• ChatCompletionMessage: Represents a single message within a conversation, including role, content, optional name, tool calls, and tool call IDs.
• ChatCompletionRequest: The full request payload including messages, model selection, sampling parameters (temperature, top_p, frequency_penalty, etc.), streaming flag, tool definitions, response format, and debug configuration.
• ChatCompletionResponseChoice / ChatCompletionStreamResponseChoice: Response choice structures for both non-streaming and streaming modes.
• ChatCompletionResponse / ChatCompletionStreamResponse: Complete response objects with ID, choices, timestamp, model name, and system fingerprint.
• GenerateUUID: A utility function to generate random alphanumeric strings for unique identifiers.

All classes that accept JSON input provide a static FromJSON factory method that returns a Result<T>, and all classes that produce JSON output provide an AsJSON method returning a picojson::object.

Usage

This header is included by the JSON FFI engine implementation to parse incoming chat completion requests and construct responses. The typical flow is:

1. A JSON string arrives from the client.
2. ChatCompletionRequest::FromJSON parses it into the structured request object.
3. The engine processes the request and produces tokens.
4. Response objects (ChatCompletionResponse or ChatCompletionStreamResponse) are constructed and serialized back to JSON via AsJSON.

Code Reference

Source Location

Property	Value
File	cpp/json_ffi/openai_api_protocol.h
Namespace	mlc::llm::json_ffi
Lines	207
Include Guard	MLC_LLM_JSON_FFI_OPENAI_API_PROTOCOL_H

Signature

namespace mlc {
namespace llm {
namespace json_ffi {

enum class Type { text, json_object, function };
enum class FinishReason { stop, length, tool_calls, error };

inline std::string GenerateUUID(size_t length);

class ChatFunction {
 public:
  std::optional<std::string> description;
  std::string name;
  std::unordered_map<std::string, std::string> parameters;
  static Result<ChatFunction> FromJSON(const picojson::object& json);
  picojson::object AsJSON() const;
};

class ChatTool {
 public:
  Type type = Type::function;
  ChatFunction function;
  static Result<ChatTool> FromJSON(const picojson::object& json);
  picojson::object AsJSON() const;
};

class ChatFunctionCall {
 public:
  std::string name;
  std::optional<std::unordered_map<std::string, std::string>> arguments;
  static Result<ChatFunctionCall> FromJSON(const picojson::object& json);
  picojson::object AsJSON() const;
};

class ChatToolCall {
 public:
  std::string id = "call_" + GenerateUUID(8);
  Type type = Type::function;
  ChatFunctionCall function;
  static Result<ChatToolCall> FromJSON(const picojson::object& json);
  picojson::object AsJSON() const;
};

class ChatCompletionMessage {
 public:
  ChatCompletionMessageContent content;
  std::string role;
  std::optional<std::string> name;
  std::optional<std::vector<ChatToolCall>> tool_calls;
  std::optional<std::string> tool_call_id;
  static Result<ChatCompletionMessage> FromJSON(const picojson::object& json);
  picojson::object AsJSON() const;
};

class ChatCompletionRequest {
 public:
  std::vector<ChatCompletionMessage> messages;
  std::optional<std::string> model;
  std::optional<double> frequency_penalty;
  std::optional<double> presence_penalty;
  bool logprobs = false;
  int top_logprobs = 0;
  std::optional<int> max_tokens;
  int n = 1;
  std::optional<int> seed;
  std::optional<std::vector<std::string>> stop;
  bool stream = false;
  std::optional<double> temperature;
  std::optional<double> top_p;
  std::optional<std::vector<ChatTool>> tools;
  std::optional<std::string> tool_choice;
  static Result<ChatCompletionRequest> FromJSON(const std::string& json_str);
};

class ChatCompletionResponse {
 public:
  std::string id;
  std::vector<ChatCompletionResponseChoice> choices;
  int created;
  std::string model;
  std::string system_fingerprint;
  std::string object = "chat.completion";
  picojson::object AsJSON() const;
};

class ChatCompletionStreamResponse {
 public:
  std::string id;
  std::vector<ChatCompletionStreamResponseChoice> choices;
  int created;
  std::string model;
  std::string system_fingerprint;
  std::string object = "chat.completion.chunk";
  std::optional<picojson::value> usage;
  picojson::object AsJSON() const;
};

}  // namespace json_ffi
}  // namespace llm
}  // namespace mlc

Import

#include "json_ffi/openai_api_protocol.h"

Dependencies:

• ctime, optional, random, string, unordered_map, vector (standard library)
• ../serve/config.h for DebugConfig and ResponseFormat
• ../support/result.h for the Result<T> type
• picojson.h for JSON parsing and serialization

I/O Contract

ChatCompletionRequest::FromJSON

Direction	Name	Type	Description
Input	json_str	const std::string&	A JSON string conforming to the OpenAI chat completion request schema
Output	(return)	Result<ChatCompletionRequest>	A parsed request object or an error result

ChatCompletionResponse::AsJSON

Direction	Name	Type	Description
Input	(this)	ChatCompletionResponse	A populated response object with id, choices, model, etc.
Output	(return)	picojson::object	A JSON object ready for serialization to string

ChatCompletionMessageContent Variant

Method	Return Type	Description
IsNull()	bool	Returns true if content holds neither text nor parts
IsText()	bool	Returns true if content is plain text
IsParts()	bool	Returns true if content is multi-part (e.g., text + image_url)
Text()	const std::string&	Returns the text value (only valid when IsText() is true)
Parts()	const std::vector<...>&	Returns the parts vector (only valid when IsParts() is true)

Usage Examples

Parsing a chat completion request:

#include "json_ffi/openai_api_protocol.h"

using namespace mlc::llm::json_ffi;

std::string raw_json = R"({"messages": [{"role": "user", "content": "Hello"}], "stream": true})";
Result<ChatCompletionRequest> result = ChatCompletionRequest::FromJSON(raw_json);
if (result.IsOk()) {
  ChatCompletionRequest request = result.Unwrap();
  // Access request.messages, request.stream, etc.
}

Constructing and serializing a response:

ChatCompletionResponse response;
response.id = "chatcmpl-" + GenerateUUID(29);
response.model = "llama-2-7b";
ChatCompletionResponseChoice choice;
choice.index = 0;
choice.finish_reason = FinishReason::stop;
choice.message.role = "assistant";
choice.message.content = ChatCompletionMessageContent("Hello! How can I help?");
response.choices.push_back(choice);

picojson::object json_obj = response.AsJSON();

Related Pages

• Mlc_ai_Mlc_llm_Engine_Interface - The serving engine that processes these requests
• Mlc_ai_Mlc_llm_Serve_Data_Header - Data structures for the serving pipeline
• Mlc_ai_Mlc_llm_Engine_Action - Engine actions that execute during request processing