Principle:Mlc ai Mlc llm Prefill Decode Disaggregation

Knowledge Sources	MLC-LLM Splitwise DistServe
Domains	Deep_Learning, Distributed_Serving
Last Updated	2026-02-09 00:00 GMT

Overview

Prefill-decode disaggregation is the technique of splitting the two phases of autoregressive LLM inference -- prompt prefill and token-by-token decode -- across separate GPU instances, transferring the computed KV cache state between them via a structured multi-step protocol.

Description

Autoregressive LLM inference consists of two fundamentally different computational phases:

• Prefill phase: The entire input prompt is processed in a single forward pass to compute attention keys and values (the KV cache) for all prompt tokens. This phase is compute-bound, with high arithmetic intensity due to parallel attention computation across the full sequence length.
• Decode phase: New tokens are generated one at a time, each requiring attention over the full KV cache built during prefill plus all previously generated tokens. This phase is memory-bandwidth-bound, as each step reads the entire KV cache but performs relatively little computation.

When both phases run on the same GPU, they interfere with each other: prefill work can delay decode steps (increasing time-to-first-token for queued requests), and decode batches can delay new prefills. Disaggregation eliminates this interference by running each phase on dedicated hardware.

The challenge is that the decode phase needs the KV cache produced by the prefill phase. MLC-LLM solves this with a three-step microserving protocol:

1. Prepare-receive (prep_recv): The decode instance allocates KV cache entries for the incoming prompt, checks its prefix cache for already-cached portions, and returns metadata describing where in GPU memory the KV data should be written.
2. Remote-send (remote_send): The prefill instance computes the KV cache for the uncached portion of the prompt, then uses NVSHMEM to write the KV data directly into the decode instance's GPU memory at the locations specified by the metadata from step 1.
3. Start-generate (start_generate): The decode instance performs any remaining local prefill (e.g., for the final token window) and begins autoregressive decode, streaming generated tokens back to the router.

This protocol supports prefix caching: if the decode instance already has some prefix of the prompt cached (from a previous request), the prefill instance only needs to compute and transfer the uncached suffix, reducing both computation and transfer time.

Usage

Use prefill-decode disaggregation when:

• You are serving long-context LLMs where the prefill phase constitutes a large fraction of total latency.
• Your deployment has multiple GPUs available and you want to specialize them for different workload characteristics.
• You need to minimize time-to-first-token by ensuring that decode instances are not blocked by prefill computation.
• Your workload exhibits prefix sharing across requests (e.g., shared system prompts), and you want to exploit prefix caching on the decode side.

Theoretical Basis

The Prefill-Decode Latency Gap

For a transformer with L layers, hidden dimension d, and h attention heads, the computational profile differs dramatically between phases:

Metric	Prefill (sequence length n)	Decode (1 token, KV cache length n)
FLOPs per layer	O(n^2 * d) for attention + O(n * d^2) for FFN	O(n * d) for attention + O(d^2) for FFN
Memory reads	Model weights (once)	Model weights + full KV cache
Arithmetic intensity	High (compute-bound)	Low (memory-bound)

Disaggregation allows each phase to be batched and scheduled independently, maximizing hardware utilization.

Three-Step Transfer Protocol

The protocol pseudocode:

disaggregated_completion(request):
    P = prefill_server
    D = decode_server (least loaded)

    # Step 1: Decode instance prepares to receive KV data
    kv_window_end = compute_split_point(request.prompt, pd_balance_factor)
    metadata, prefix_len = D.prep_recv(request.prompt[0:kv_window_end])

    # Step 2: Prefill instance computes and transfers uncached KV
    if prefix_len < kv_window_end:
        P.remote_send(
            prompt = request.prompt,
            range  = [prefix_len, kv_window_end],
            dest   = metadata,   # GPU memory addresses on D
            rank   = D.device_id_start
        )

    # Step 3: Decode instance starts generating
    yield from D.start_generate(
        prompt = request.prompt,
        begin  = kv_window_end    # local prefill range: [kv_window_end:]
    )

PD Balance Factor

The pd_balance_factor parameter controls how much of the prompt is handled by the decode instance locally versus being prefilled on the prefill instance and transferred:

• pd_balance_factor = 0.0: The prefill instance handles the full prompt (kv_window_end = len(prompt) - 1). The decode instance only processes the last token locally.
• pd_balance_factor = 0.5: The prompt is split roughly in half between prefill transfer and local decode prefill.
• Higher values shift more work to the decode instance, reducing transfer volume but potentially increasing decode-side latency.

The split point is computed as:

kv_window_end = int((1 - pd_balance_factor) * len(prompt))

Prefix Cache Optimization

When the decode instance has prefix caching enabled (radix tree), step 1 returns the length of the longest cached prefix. The prefill instance then only needs to compute KV entries for prompt[prefix_matched_length:kv_window_end], which can be a significant reduction for workloads with shared system prompts or repeated prefixes.

Related Pages

Implemented By

• Implementation:Mlc_ai_Mlc_llm_Microserving_Endpoints