Implementation:Mlc ai Mlc llm JSONFFIEngine Header

Knowledge Sources	Mlc_ai_Mlc_llm
Domains	C++, FFI, LLM Inference, Serving
Last Updated	2026-02-09 19:00 GMT

Overview

json_ffi_engine.h declares the JSONFFIEngine C++ class, which is the core engine interface for processing JSON-formatted chat completion requests in MLC LLM, bridging the FFI boundary between client code and the threaded serving engine.

Description

The JSONFFIEngine class, defined within the mlc::llm::json_ffi namespace, provides the C++ interface for handling LLM inference requests formatted as JSON strings following the OpenAI chat completion protocol. It wraps a ThreadedEngine and manages tokenization, conversation templates, and request state.

Public interface:

• JSONFFIEngine() / ~JSONFFIEngine() -- Constructor and destructor for engine lifecycle management
• ChatCompletion(request_json_str, request_id) -- Processes a chat completion request. Accepts a JSON string and a unique request ID. Returns bool indicating success.
• AddRequest(request_json_str, request_id) -- Adds a new request to the engine queue. Returns bool indicating success.
• StreamBackError(request_id) -- Streams back an error for a specific request ID.
• Abort(request_id) -- Aborts a running request. Returns bool indicating success.
• GetLastError() -- Returns the last error message as a string.
• ExitBackgroundLoop() -- Signals the background processing loop to exit.

Protected members:

• RequestState -- An inner struct representing the local state for each reply stream, containing:
  • model -- The model name to include in the reply
  • streamer -- A vector of TextStreamer instances for token-to-text conversion per stream
• engine_ -- A std::unique_ptr<ThreadedEngine> that handles the actual model inference on background threads
• err_ -- A std::string holding the last error message
• request_stream_callback_ -- A TVM Function that serves as the callback for streaming responses back to the client
• tokenizer_ -- The tokenizer used for encoding/decoding text
• conv_template_ -- The Conversation template that defines prompt formatting
• default_generation_config_ -- Default generation parameters (temperature, top_p, etc.)
• model_config_ -- The ModelConfig containing model structural parameters
• device_ -- The DLDevice on which inference runs
• request_map_ -- An unordered_map from request ID (String) to RequestState, tracking all active requests

Usage

This header is included by the JSON FFI engine implementation file and by the TVM module registration code that exposes the engine to foreign language runtimes (Java, Kotlin, Swift, etc.). It is the central interface through which all chat completion requests flow in the MLC LLM system.

Code Reference

Source Location

• Repository: Mlc_ai_Mlc_llm
• File: cpp/json_ffi/json_ffi_engine.h

Signature

namespace mlc {
namespace llm {
namespace json_ffi {

class JSONFFIEngine {
 public:
  JSONFFIEngine();
  ~JSONFFIEngine();

  bool ChatCompletion(std::string request_json_str, std::string request_id);
  bool AddRequest(std::string request_json_str, std::string request_id);
  void StreamBackError(std::string request_id);
  bool Abort(std::string request_id);
  std::string GetLastError();
  void ExitBackgroundLoop();

 protected:
  struct RequestState {
    std::string model;
    std::vector<TextStreamer> streamer;
  };

  std::unique_ptr<ThreadedEngine> engine_;
  std::string err_;
  Function request_stream_callback_;
  Tokenizer tokenizer_;
  Conversation conv_template_;
  GenerationConfig default_generation_config_;
  ModelConfig model_config_;
  DLDevice device_;
  std::unordered_map<String, RequestState> request_map_;
};

}  // namespace json_ffi
}  // namespace llm
}  // namespace mlc

Import

#include "json_ffi_engine.h"

I/O Contract

Method	Input	Type	Description
ChatCompletion	request_json_str	std::string	JSON string following OpenAI chat completion request format
ChatCompletion	request_id	std::string	Unique identifier for tracking the request
AddRequest	request_json_str	std::string	JSON string of the request to add
AddRequest	request_id	std::string	Unique request identifier
StreamBackError	request_id	std::string	ID of the request to error
Abort	request_id	std::string	ID of the request to abort

Method	Return Type	Description
ChatCompletion	bool	true on success; on failure, error is stored and retrievable via GetLastError()
AddRequest	bool	true on success
Abort	bool	true if the request was successfully aborted
GetLastError	std::string	The last error message encountered

Protected Member	Type	Purpose
engine_	unique_ptr<ThreadedEngine>	Runs model inference on background threads
tokenizer_	Tokenizer	Encodes text to tokens and decodes tokens back to text
conv_template_	Conversation	Defines prompt formatting rules for the loaded model
default_generation_config_	GenerationConfig	Default sampling parameters for generation
model_config_	ModelConfig	Model architecture configuration (vocab size, context window, etc.)
request_map_	unordered_map<String, RequestState>	Maps request IDs to their per-stream state

Usage Examples

#include "json_ffi_engine.h"

// The engine is typically created and exposed via TVM module registration.
// Client code interacts through the registered functions:

JSONFFIEngine engine;

// Process a chat completion request
std::string request_json = R"({
    "messages": [{"role": "user", "content": "Hello!"}],
    "model": "Llama-3-8B-q4f16_1",
    "stream": true
})";

bool success = engine.ChatCompletion(request_json, "req-001");
if (!success) {
    std::string error = engine.GetLastError();
    LOG(ERROR) << "ChatCompletion failed: " << error;
}

// Abort a request
engine.Abort("req-001");

// Shut down the background loop
engine.ExitBackgroundLoop();