Implementation:Mlc ai Mlc llm Router Handle completion

Knowledge Sources	MLC-LLM
Domains	Deep_Learning, Distributed_Serving
Last Updated	2026-02-09 00:00 GMT

Overview

Concrete tool for handling request preemption in distributed inference systems with automatic retry loops, provided by MLC-LLM.

Description

Router.handle_completion is the top-level method that processes a completion request through the routing pipeline with built-in preemption recovery. It performs three responsibilities:

1. Prompt tokenization: If the request's prompt field is a string, it encodes it into a list of token IDs using the router's tokenizer. This ensures that all downstream processing works with tokenized prompts.
2. Debug configuration: If the request lacks a debug_config, it initializes one with default values. This is required because the microserving endpoints use the disagg_config sub-field of debug_config to communicate disaggregation parameters to the engine.
3. Retry loop: It calls translate_request in a while not completed loop. If translate_request yields None (signaling preemption), the loop restarts the entire translation from scratch. Valid response chunks are yielded through to the caller. The loop exits only when translate_request completes without yielding None.

This method is the entry point called by the serve() function's /v1/completions endpoint handler.

Usage

Use handle_completion when processing completion requests through the router. It is the primary interface between the HTTP API layer and the routing logic. It should be preferred over calling translate_request directly because it provides tokenization, debug config setup, and preemption recovery that translate_request alone does not provide.

Code Reference

Source Location

• Repository: MLC-LLM
• File: python/mlc_llm/router/router.py (Lines 111-131)

Signature

async def handle_completion(
    self,
    request: openai_api_protocol.CompletionRequest,
    request_id: str,
) -> AsyncGenerator[openai_api_protocol.CompletionResponse, Any]:

Import

from mlc_llm.router import Router

# handle_completion is an instance method on Router
router = Router(model="your-model", ...)
# Called as: router.handle_completion(request, request_id)

I/O Contract

Inputs

Name	Type	Required	Description
request	openai_api_protocol.CompletionRequest	Yes	The OpenAI-compatible completion request. The prompt field can be either a str (which will be tokenized) or a List[int] (pre-tokenized). Other fields (max_tokens, stream, temperature, etc.) are passed through to the underlying engines.
request_id	str	Yes	A unique identifier for this request (e.g., "cmpl-{uuid}"). Used for tracking across the microserving protocol and for logging.

Outputs

Name	Type	Description
(yields)	AsyncGenerator[openai_api_protocol.CompletionResponse, Any]	An async generator yielding CompletionResponse objects. In streaming mode, multiple chunks are yielded as tokens are generated. In non-streaming mode, a single response is yielded. Preemption is handled internally; callers never see None values. The generator completes only after the request has been fully served (possibly after one or more transparent retries).

Usage Examples

Basic Usage

from mlc_llm.router import Router
from mlc_llm.protocol.openai_api_protocol import CompletionRequest
from mlc_llm.serve.engine_utils import random_uuid

router = Router(
    model="dist/Llama-2-7b-chat-hf-q4f16_1-MLC",
    hosts=["127.0.0.1", "127.0.0.1"],
    ports=[8080, 8081],
    num_gpus=[1, 1],
    router_mode="disagg",
)

request = CompletionRequest(
    model="Llama-2-7b-chat",
    prompt="Once upon a time",
    max_tokens=100,
    stream=True,
)
request_id = f"cmpl-{random_uuid()}"

# The async generator handles preemption transparently.
# If the engine preempts this request, handle_completion
# automatically retries without the caller needing to
# implement any retry logic.
async for response in router.handle_completion(request, request_id):
    for choice in response.choices:
        print(choice.text, end="", flush=True)
print()  # newline after completion

Non-Streaming with Preemption Resilience

from mlc_llm.router import Router
from mlc_llm.protocol.openai_api_protocol import CompletionRequest

router = Router(
    model="dist/Llama-2-7b-chat-hf-q4f16_1-MLC",
    hosts=["127.0.0.1", "127.0.0.1"],
    ports=[8080, 8081],
    num_gpus=[1, 1],
    router_mode="disagg",
)

request = CompletionRequest(
    model="Llama-2-7b-chat",
    prompt="Explain quantum computing in one paragraph.",
    max_tokens=200,
    stream=False,
)
request_id = "cmpl-example-001"

# Even in non-streaming mode, the retry loop is active.
# If the request is preempted, it will be silently retried
# until a complete response is obtained.
output_text = ""
async for response in router.handle_completion(request, request_id):
    for choice in response.choices:
        output_text += choice.text

print(output_text)

Related Pages

Implements Principle

• Principle:Mlc_ai_Mlc_llm_Preemption_Recovery