Implementation:Mlc ai Mlc llm Microserving Endpoints

Knowledge Sources	MLC-LLM
Domains	Deep_Learning, Distributed_Serving
Last Updated	2026-02-09 00:00 GMT

Overview

Concrete tool for disaggregating the prefill and decode phases of autoregressive LLM inference across separate GPU instances, provided by MLC-LLM.

Description

The microserving endpoints module exposes three FastAPI HTTP endpoints that implement the three-step disaggregated serving protocol. Each endpoint is registered on a per-engine server instance and translates microserving-specific request parameters into the engine's internal DisaggConfig before delegating to the standard request_completion handler:

• /microserving/prep_recv: Invoked on the decode instance. Allocates KV cache entries for the incoming prompt, checks the prefix cache for already-cached portions, and returns metadata (KV append addresses and prefix match length) that the prefill instance needs to perform direct GPU memory writes.
• /microserving/remote_send: Invoked on the prefill instance. Computes the KV cache for the specified prompt range and uses NVSHMEM to transfer the data directly into the decode instance's GPU memory at the addresses provided by prep_recv. Returns an empty acknowledgment on completion.
• /microserving/start_generate: Invoked on the decode instance after KV transfer is complete. Performs any remaining local prefill for the tail of the prompt and begins autoregressive decode, returning generated tokens as a streaming or non-streaming completion response.

All three endpoints internally set the appropriate DisaggConfig on the request's debug_config field, which instructs the engine to execute the correct disaggregated operation rather than a standard completion.

Usage

These endpoints are not called directly by users. They are invoked by the Router class during disaggregated request handling. Each MLC-LLM engine server automatically registers these endpoints when started in server mode. The router orchestrates the three-step protocol by making HTTP POST requests to these endpoints on the appropriate engine instances.

Code Reference

Source Location

• Repository: MLC-LLM
• File: python/mlc_llm/serve/entrypoints/microserving_entrypoints.py (Lines 22-73)
• Protocol definitions: python/mlc_llm/protocol/microserving_protocol.py

Signatures

Endpoint 1: prep_recv

@app.post("/microserving/prep_recv")
async def prep_recv(
    request: PrepRecvRequest,
    raw_request: fastapi.Request,
) -> PrepRecvResponse:

Endpoint 2: remote_send

@app.post("/microserving/remote_send")
async def remote_send(
    request: RemoteSendRequest,
    raw_request: fastapi.Request,
) -> dict:

Endpoint 3: start_generate

@app.post("/microserving/start_generate")
async def start_generate(
    request: StartGenerateRequest,
    raw_request: fastapi.Request,
) -> Response:

Import

# The endpoints are registered as a FastAPI APIRouter
from mlc_llm.serve.entrypoints.microserving_entrypoints import app

# Protocol models used by the endpoints
from mlc_llm.protocol.microserving_protocol import (
    PrepRecvRequest,
    PrepRecvResponse,
    RemoteSendRequest,
    StartGenerateRequest,
)

I/O Contract

Endpoint 1: prep_recv

Inputs:

Name	Type	Required	Description
request	PrepRecvRequest	Yes	Extends CompletionRequest with an additional end field. Contains the full prompt and the KV window end index. The end field specifies the upper bound of the KV range ([0, end]) to allocate on the decode instance. Supports Python-style negative indexing (e.g., -1 means len(prompt) - 1).
raw_request	fastapi.Request	Yes	The raw FastAPI request object, passed through to the engine for disconnection detection.

Outputs:

Name	Type	Description
(response)	PrepRecvResponse	A Pydantic model containing two fields: kv_append_metadata (a base64-encoded string describing the GPU memory addresses where KV data should be written) and prefix_matched_length (an integer indicating how many tokens of the prompt are already cached on the decode instance).

Endpoint 2: remote_send

Inputs:

Name	Type	Required	Description
request	RemoteSendRequest	Yes	Extends CompletionRequest with four additional fields: begin (start of the KV range to compute), end (end of the KV range to compute), kv_addr_info (base64-encoded metadata from prep_recv specifying destination GPU memory addresses), and recv_rank (the NVSHMEM PE offset of the destination decode instance).
raw_request	fastapi.Request	Yes	The raw FastAPI request object.

Outputs:

Name	Type	Description
(response)	dict	An empty dictionary ({}) serving as an acknowledgment that the prefill computation and KV transfer have completed.

Endpoint 3: start_generate

Inputs:

Name	Type	Required	Description
request	StartGenerateRequest	Yes	Extends CompletionRequest with an additional begin field specifying the start of the KV range for local prefill on the decode instance. The range [begin:] will be prefilled locally before decode begins.
raw_request	fastapi.Request	Yes	The raw FastAPI request object.

Outputs:

Name	Type	Description
(response)	Response	A standard completion response (streaming or non-streaming) as returned by request_completion. In streaming mode, this is a server-sent events stream of CompletionResponse chunks. In non-streaming mode, this is a single JSON CompletionResponse.

Usage Examples

Basic Usage

import aiohttp
import json

# These endpoints are called by the Router, not directly by users.
# This example shows the HTTP-level interaction for educational purposes.

async def disaggregated_completion(prompt_tokens, prefill_url, decode_url):
    headers = {"Content-Type": "application/json"}
    async with aiohttp.ClientSession() as session:
        # Step 1: Ask decode instance to prepare for receiving KV data
        prep_payload = {
            "prompt": prompt_tokens,
            "end": -1,  # full prompt minus last token
            "stream": False,
        }
        async with session.post(
            f"{decode_url}/microserving/prep_recv",
            json=prep_payload,
            headers=headers,
        ) as resp:
            data = await resp.json()
            kv_metadata = data["kv_append_metadata"]
            prefix_len = data["prefix_matched_length"]

        # Step 2: Ask prefill instance to compute and transfer KV
        kv_window_end = len(prompt_tokens) - 1
        if prefix_len < kv_window_end:
            send_payload = {
                "prompt": prompt_tokens,
                "begin": prefix_len,
                "end": kv_window_end,
                "kv_addr_info": kv_metadata,
                "recv_rank": 1,  # decode instance PE offset
                "stream": False,
            }
            async with session.post(
                f"{prefill_url}/microserving/remote_send",
                json=send_payload,
                headers=headers,
            ) as resp:
                await resp.json()  # wait for ack

        # Step 3: Ask decode instance to start generating
        gen_payload = {
            "prompt": prompt_tokens,
            "begin": kv_window_end,
            "stream": True,
            "max_tokens": 100,
        }
        async with session.post(
            f"{decode_url}/microserving/start_generate",
            json=gen_payload,
            headers=headers,
        ) as resp:
            async for chunk in resp.content:
                chunk = chunk.strip()
                if chunk and chunk != b"\n":
                    raw = chunk[6:].strip()
                    if raw != b"[DONE]":
                        print(json.loads(raw))

Related Pages

Implements Principle

• Principle:Mlc_ai_Mlc_llm_Prefill_Decode_Disaggregation