Implementation:Mlc ai Mlc llm Event Trace Recorder

Overview

File: cpp/serve/event_trace_recorder.cc

Purpose: Provides the concrete implementation of the EventTraceRecorder interface. This class records timestamped events for each request processed by the serving engine and can export them in the Chrome Trace Event Format (JSON), enabling performance analysis and visualization in tools like chrome://tracing or Perfetto.

Namespace: mlc::llm::serve

Helper: detail::PairHash

struct PairHash {
  template <class T1, class T2>
  std::size_t operator()(const std::pair<T1, T2>& p) const {
    auto h1 = std::hash<T1>{}(p.first);
    auto h2 = std::hash<T2>{}(p.second);
    return h1 ^ h2;
  }
};

A hash functor for std::pair keys, used by the event_counter_ map to count occurrences of each (request_id, event_name) pair. Uses XOR composition of individual element hashes.

Class: EventTraceRecorderImpl

Inherits from EventTraceRecorderObj and provides thread-safe event recording.

AddEvent (Single Request)

void AddEvent(const String& request_id, const std::string& event) final;

Records a timestamped event for a single request. Captures the current system time as a high-resolution double (seconds since epoch), acquires the mutex, and delegates to AddEventInternal.

AddEvent (Multiple Requests)

void AddEvent(const Array<String>& request_ids, const std::string& event) final;

Records the same event (with the same timestamp) for multiple requests. This is used when a batched operation affects multiple requests simultaneously. A single timestamp is captured before the lock, ensuring all requests in the batch share the exact same event time.

DumpJSON

std::string DumpJSON() final;

Exports all recorded events as a JSON string in the Chrome Trace Event Format. The method:

1. Takes a snapshot of the events map under the mutex lock.
2. Iterates over requests in their original submission order (via request_id_in_order_).
3. Classifies each event string into one of three Chrome Trace phases:
   • Events starting with "start " become duration begin events (phase "B").
   • Events starting with "finish " become duration end events (phase "E").
   • All other events become instant events (phase "i").
4. Converts timestamps from seconds to microseconds (multiplied by 1e6).
5. Constructs JSON objects with fields: name, ph (phase), ts (timestamp), pid (fixed at 1), and tid (set to the request ID for visual grouping per request).
6. Sorts events by timestamp within each request.
7. Serializes the complete array to a JSON string using picojson.

The output format example:

[
  {"name": "prefill", "ph": "B", "ts": 1700000000000000, "pid": 1, "tid": "req-001"},
  {"name": "prefill", "ph": "E", "ts": 1700000000050000, "pid": 1, "tid": "req-001"}
]

AddEventInternal

void AddEventInternal(const std::string& request_id, const std::string& event,
                      double event_time);

Internal implementation of event recording (must be called under the mutex). Performs:

1. Adds the request ID to request_id_in_order_ if it has not been seen before.
2. Increments the event counter for the (request_id, event) pair.
3. Stores the event with an appended occurrence count (e.g., "start prefill (0)", "start prefill (1)") along with its timestamp.

The occurrence count allows multiple invocations of the same event for a single request to be distinguished in the trace output.

Thread Safety

Member	Protection
mutex_	std::mutex guarding all critical regions
request_id_in_order_	Protected -- records request IDs in submission order
event_counter_	Protected -- counts occurrences of each (request_id, event) pair
events_	Protected -- stores all event data with timestamps

All public AddEvent methods acquire the mutex before modifying shared state. The DumpJSON method takes a snapshot under the lock and then processes it without holding the lock, minimizing contention.

Factory and Registration

EventTraceRecorder EventTraceRecorder::Create() {
  return EventTraceRecorder(tvm::ffi::make_object<EventTraceRecorderImpl>());
}

Static factory method that creates a new EventTraceRecorderImpl instance.

The TVM FFI registration block exposes three functions:

• mlc.serve.EventTraceRecorder: Constructor, returns a new recorder.
• mlc.serve.EventTraceRecorderAddEvent: Adds an event for a single request.
• mlc.serve.EventTraceRecorderDumpJSON: Dumps all events as JSON.

TVM_FFI_STATIC_INIT_BLOCK() {
  namespace refl = tvm::ffi::reflection;
  EventTraceRecorderImpl::RegisterReflection();
  refl::GlobalDef()
      .def("mlc.serve.EventTraceRecorder", []() { return EventTraceRecorder::Create(); })
      .def("mlc.serve.EventTraceRecorderAddEvent",
           [](const EventTraceRecorder& trace_recorder, const String& request_id,
              const std::string& event) { trace_recorder->AddEvent(request_id, event); })
      .def_method("mlc.serve.EventTraceRecorderDumpJSON", &EventTraceRecorderObj::DumpJSON);
}

Design Notes

• The Chrome Trace Event Format was chosen because it is widely supported by browser-based profiling tools and provides a familiar visualization for duration and instant events.
• Using pid = 1 for all events and tid = request_id means the trace viewer will display one "thread" lane per request, making it easy to visualize the timeline of each request independently.
• The event occurrence counter ensures that repeated events (e.g., multiple prefill rounds for chunked prefill) are distinguishable in the trace output.