Principle:Mlc ai Mlc llm Multi Engine Infrastructure
| Knowledge Sources | |
|---|---|
| Domains | Deep_Learning, Distributed_Serving |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
Multi-engine infrastructure is the practice of deploying multiple independent LLM inference engine instances across separate GPU resources, each serving as a specialized endpoint that can be coordinated by a central router to perform different phases of autoregressive generation.
Description
In large-scale LLM serving, a single monolithic engine may not efficiently handle the diverse computational demands of different inference phases. The prefill phase (processing the full input prompt) is compute-bound and benefits from high parallelism, while the decode phase (generating tokens one at a time) is memory-bandwidth-bound and benefits from dedicated memory resources. Multi-engine infrastructure addresses this by spawning multiple engine processes, each bound to a specific set of GPUs, and coordinating them through a shared communication layer.
The infrastructure layer is responsible for:
- Spawning engine processes: Each engine is started as an independent server (using subprocess management such as
PopenServer) bound to specific GPU device IDs and network ports. - Establishing GPU-to-GPU communication: NVSHMEM (NVIDIA Symmetric Hierarchical Memory) is initialized with a shared unique identifier (UID) so that all engines can perform direct GPU memory transfers for KV cache migration without routing data through the CPU or network stack.
- Assigning device partitions: A mapping from engine index to GPU device IDs is computed (via cumulative sums of per-endpoint GPU counts), ensuring no two engines share the same GPU and that the NVSHMEM process element (PE) indices are correctly assigned.
- Tracking endpoint load: A per-endpoint running request counter enables the router to make load-aware scheduling decisions when dispatching requests.
This architecture forms the foundation for disaggregated serving, where prefill and decode workloads are physically separated to achieve higher throughput and lower latency compared to co-located execution.
Usage
Use multi-engine infrastructure when:
- You need to serve LLM inference at scale with multiple GPUs and want to specialize different GPUs for different inference phases.
- Your workload benefits from separating prefill (prompt processing) from decode (token generation) across dedicated hardware.
- You want to enable KV cache transfer between engines via high-speed GPU interconnects (NVLink or NVSwitch) using NVSHMEM.
- You are building a disaggregated serving system where a router dispatches requests to multiple backend engines.
Theoretical Basis
Disaggregated Serving Architecture
The theoretical foundation comes from the observation that prefill and decode phases have fundamentally different hardware utilization profiles:
| Phase | Compute Characteristic | Bottleneck | Optimal Hardware |
|---|---|---|---|
| Prefill | High arithmetic intensity (parallel attention over full prompt) | Compute (FLOPs) | High-FLOPS GPUs, tensor parallelism |
| Decode | Low arithmetic intensity (single token at a time) | Memory bandwidth | High-bandwidth memory, batching |
By deploying separate engine instances for each phase, the system can:
- Independently scale prefill and decode capacity based on workload characteristics.
- Avoid interference between compute-bound prefill and memory-bound decode on the same GPU.
- Use GPU-direct memory transfers (NVSHMEM) to migrate KV cache state between prefill and decode engines with minimal latency.
NVSHMEM Initialization
The shared memory layer requires a globally unique identifier (UID) that all participating processes use to join the same NVSHMEM "world." The initialization sequence is:
1. Router calls runtime.disco.nvshmem.init_nvshmem_uid() to generate a UID.
2. For each engine i with device_id_start[i] and num_gpus[i]:
- pe_start = device_id_start[i]
- npes = total_gpus (across all engines)
3. Each engine receives the UID and PE configuration via environment variable
MLC_NVSHMEM_INIT_CONFIG_JSON_STR.
4. Engines join the NVSHMEM world concurrently (required, since initialization
is a collective operation across all PEs).
This ensures that when a prefill engine computes KV cache entries, it can write them directly into the memory of a decode engine's GPU without any CPU-mediated data movement.
Load-Aware Endpoint Selection
The infrastructure tracks per-endpoint request counts and uses a least-loaded selection policy:
pick_endpoint(candidate_ids):
best = argmin(num_running_requests[id] for id in candidate_ids)
return best
This simple greedy policy balances load across decode endpoints, preventing any single engine from becoming a bottleneck.