Principle:Mlc ai Mlc llm Preemption Recovery

Knowledge Sources	MLC-LLM Splitwise DistServe
Domains	Deep_Learning, Distributed_Serving
Last Updated	2026-02-09 00:00 GMT

Overview

Preemption recovery is the mechanism by which a distributed inference system detects that an in-flight request has been interrupted due to resource pressure, and automatically retries the request from scratch to ensure eventual completion without requiring client-side retry logic.

Description

In high-throughput LLM serving, engine instances manage finite GPU memory for the KV cache. When memory pressure is high (e.g., many concurrent requests or long sequences), the engine scheduler may need to evict (preempt) an in-progress request to free KV cache space for higher-priority or more urgent work. This preemption is communicated back to the caller as a special "preempt" finish reason in the completion response.

Preemption recovery addresses this by implementing a transparent retry loop at the router level:

• Detection: When the router receives a response chunk with finish_reason == "preempt", it recognizes that the request was not completed normally. The translation layer signals this by yielding None instead of a valid response.
• Retry: The outer completion handler interprets the None signal as an instruction to restart the entire request translation from scratch. This means the full disaggregated protocol (prepare-receive, remote-send, start-generate) or round-robin dispatch is re-executed.
• Transparency: The retry loop is invisible to the end user. The client receives a continuous stream of valid response chunks, with any preempted attempts silently discarded and restarted.

This pattern is essential in disaggregated serving because preemption on the decode instance means that the KV cache entries allocated during prep_recv have been invalidated. The entire three-step protocol must be re-executed from step 1 to reallocate fresh KV cache entries.

Usage

Use preemption recovery when:

• You are building or deploying an LLM serving system that may experience memory pressure under high concurrency.
• Your engine supports KV cache eviction (preemption) as a scheduling mechanism.
• You want to shield clients from transient resource exhaustion failures, providing an "eventually complete" guarantee.
• You are using disaggregated serving where preemption on the decode side invalidates transferred KV cache state and requires full protocol restart.

Theoretical Basis

Preemption in LLM Serving

LLM engines typically use a paged KV cache (inspired by virtual memory paging) to manage GPU memory. When all KV pages are occupied, the scheduler has two options:

1. Block: Refuse new requests until existing ones complete (head-of-line blocking).
2. Preempt: Evict the KV cache of a lower-priority request, freeing pages for a new or higher-priority request. The evicted request must be restarted or resumed later.

Preemption enables the engine to maintain high throughput under bursty workloads by avoiding the queueing delays of blocking. However, it requires the routing layer to handle the resulting incomplete requests.

Retry Loop Architecture

The retry mechanism is structured as a nested loop:

handle_completion(request, request_id):
    tokenize(request.prompt)
    completed = False
    while not completed:
        completed = True
        for response in translate_request(request, request_id):
            if response is None:
                # Preemption detected -- restart from scratch
                completed = False
                break
            yield response

Key properties of this design:

• Stateless retry: Each retry re-executes the full translation, including endpoint selection. This means the retried request may be routed to a different decode instance that has available capacity.
• No partial output: When preemption occurs during streaming, any partial output from the preempted attempt is discarded. The retry generates the response from scratch. This is correct because preemption invalidates the KV cache, so partial output cannot be continued.
• Unbounded retries: The loop continues until the request completes without preemption. In practice, the engine's scheduler ensures forward progress by eventually prioritizing the retried request.

Preemption Signal Propagation

The preemption signal flows through multiple layers:

Engine scheduler (evicts KV cache)
  -> Engine response: finish_reason = "preempt"
    -> Microserving endpoint / round-robin handler: detects "preempt"
      -> translate_request: yields None
        -> handle_completion: breaks inner loop, sets completed = False
          -> Retry: re-enters translate_request

Both the disaggregated and round-robin code paths check for the "preempt" finish reason and yield None accordingly, ensuring that the retry mechanism works uniformly regardless of the routing strategy.

Impact on Disaggregated Serving

Preemption is particularly important in disaggregated serving because:

• The decode instance is typically memory-constrained (it holds KV caches for many concurrent decode requests).
• A preempted decode request means the KV cache slots allocated in step 1 (prep_recv) are invalidated.
• The retry must re-execute from step 1 to obtain fresh KV cache allocations, potentially on a different (less loaded) decode instance.

Related Pages

Implemented By

• Implementation:Mlc_ai_Mlc_llm_Router_Handle_completion