Implementation:Mlc ai Mlc llm Metrics Header

Overview

The file cpp/serve/metrics.h declares all runtime metric structures for the MLC-LLM serving engine. It provides a centralized location for metrics definitions spanning individual requests, speculative decoding statistics, time-cost tracking, and engine-wide aggregates. The header is designed so that update functions are defined inline for efficient inlining by the compiler, while the JSON serialization methods are implemented separately in metrics.cc.

File Location

cpp/serve/metrics.h

Dependencies

Header	Purpose
`picojson.h`	Lightweight JSON serialization library
`tvm/runtime/logging.h`	TVM assertion macros (`ICHECK_GE`, etc.)
`<chrono>`	High-resolution time points for latency measurement
`<string>`	String return types for JSON serialization

Namespace

All types are defined in mlc::llm::serve.

Design Principles

The header comment explicitly states the design intent:

// We keep all metrics containers in this header (instead of in Engine and Request State)
// so we have a single central place to define all metrics across the engine.
// Conceptually, these statistics are derived from engine/request behaviors.

Additionally, inline update functions are preferred over out-of-line implementations for performance:

// NOTE: we keep most update function in header
// so they can be inlined effectively

Struct: TimeCost

A utility struct for tracking the mean time cost of an operation with warmup support.

struct TimeCost {
  double sum = 0.0;
  int64_t count = 0;
  bool warmed_up = false;

  void Update(double value) {
    if (warmed_up) {
      sum += value;
      count += 1;
    } else {
      warmed_up = true;
    }
  }

  void Reset() {
    this->sum = 0.0;
    this->count = 0;
  }

  picojson::object AsJSON() const;
};

Field	Type	Description
`sum`	`double`	Accumulated cost excluding warmup
`count`	`int64_t`	Number of tracked events excluding warmup
`warmed_up`	`bool`	Whether the first (warmup) invocation has been discarded

The warmup mechanism discards the first measurement, which is typically an outlier due to JIT compilation or cache population. The Reset() method does not reset warmed_up, since subsequent resets measure the same operation.

Struct: SpecDecodeMetrics

Tracks speculative decoding acceptance statistics at each draft step.

struct SpecDecodeMetrics {
  std::vector<int64_t> draft_count;
  std::vector<int64_t> accept_count;

  void Update(int draft_length, int accept_length);
  bool IsEmpty() const { return draft_count.size() == 0; }
  void Reset();
  picojson::object AsJSON() const;
};

Field	Type	Description
`draft_count`	`std::vector<int64_t>`	Number of draft tokens proposed at each speculation step
`accept_count`	`std::vector<int64_t>`	Number of tokens accepted at each speculation step

Update Method

void Update(int draft_length, int accept_length) {
    ICHECK_GE(accept_length, 1);
    if (accept_count.size() < draft_length) {
      this->accept_count.resize(draft_length, 0);
      this->draft_count.resize(draft_length, 0);
    }
    for (int j = 0; j < draft_length; ++j) {
      if (j < accept_length) {
        ++this->accept_count[j];
      }
      ++this->draft_count[j];
    }
}

The method dynamically grows the vectors when draft_length exceeds the current size. For each step j, the draft count is always incremented, while the accept count is only incremented if step j was within the acceptance window. The assertion enforces that at least one token is always accepted.

Struct: RequestMetrics

Per-request metrics capturing token counts and timing information.

Fields

Field	Type	Description
`prompt_tokens`	`int64_t`	Number of input tokens
`completion_tokens`	`int64_t`	Number of output tokens
`prefill_tokens`	`int64_t`	Number of tokens processed during prefill
`decode_tokens`	`int64_t`	Number of tokens processed during decode (including rollbacks)
`jump_forward_tokens`	`int64_t`	Tokens predicted via jump-forward decoding
`add_time_point`	`chrono::high_resolution_clock::time_point`	Time when request was added to the engine
`prefill_end_time_point`	`chrono::high_resolution_clock::time_point`	Time when prefill stage completed
`finish_time_point`	`chrono::high_resolution_clock::time_point`	Time when all decoding finished

Timing Methods

double GetPrefillTime() const;   // prefill_end - add_time
double GetDecodeTime() const;    // finish - prefill_end
double GetTTFT() const;          // prefill_end - add_time (same as prefill time)
double GetTotalTime() const;     // finish - add_time
double GetInterTokenLatency() const; // total_time / completion_tokens

All timing methods convert from nanosecond precision chrono durations to seconds by dividing by 1e9.

The method IsComplete() returns true when both prompt_tokens and completion_tokens are non-zero, indicating a fully processed request.

Struct: EngineMetrics

Engine-wide runtime metrics aggregating across all requests and tracking batch-level timing.

Aggregate Fields

Field	Type	Description
`engine_prefill_time_sum`	`double`	Total engine time spent on prefill
`engine_decode_time_sum`	`double`	Total engine time on decode/draft/verify
`engine_jump_forward_time_sum`	`double`	Total engine time on jump-forward prediction
`prompt_tokens_sum`	`int64_t`	Total input tokens across all requests
`completion_tokens_sum`	`int64_t`	Total output tokens across all requests
`prefill_tokens_sum`	`int64_t`	Total prefill tokens (excluding prefix-cached)
`decode_tokens_sum`	`int64_t`	Total decode tokens (including rollbacks)
`jump_forward_tokens_sum`	`int64_t`	Total jump-forward tokens
`last_finished_request`	`RequestMetrics`	Metrics snapshot from the most recently finished request
`spec_decode`	`SpecDecodeMetrics`	Aggregated speculative decoding metrics

Batch-Size Disaggregated Timing

static constexpr const int64_t kEndFineGrainedTrackingBatchSize = 65;
std::vector<TimeCost> decode_time_by_batch_size;
std::vector<TimeCost> draft_time_by_batch_size;
std::vector<TimeCost> verify_time_by_batch_size;

The engine tracks decode, draft, and verify times disaggregated by batch size, up to batch size 64 (index 0 through 64, with index 0 unused). This enables performance analysis of how throughput varies with concurrent request count.

Update Methods

void UpdateDecodeTimeByBatchSize(int batch_size, double time);
void UpdateDraftTimeByBatchSize(int batch_size, double time);
void UpdateVerifyTimeByBatchSize(int effective_batch_size, double time);
void RequestFinishUpdate(const RequestMetrics& request_metrics);

The batch-size update methods silently ignore entries where batch_size >= kEndFineGrainedTrackingBatchSize. The RequestFinishUpdate method accumulates per-request metrics into the engine totals and saves the most recent completed request's metrics.

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment