Principle:FMInference FlexLLMGen HuggingFace DeepSpeed Baseline Inference
| Knowledge Sources | |
|---|---|
| Domains | LLM Inference, Benchmarking, Memory Management |
| Last Updated | 2026-02-09 12:00 GMT |
Overview
Baseline inference strategies for large language models that use weight offloading across a memory hierarchy to serve models larger than GPU memory.
Description
When running inference on large language models whose parameters exceed GPU memory capacity, two established baseline approaches exist. The first is accelerator-based dispatch, where a device map assigns individual model layers or weight tensors to specific devices (GPU, CPU, or disk). During inference, weights are loaded on-demand from their assigned device into GPU memory for computation and then discarded. The second is ZeRO Stage-3 inference, which partitions parameters across available devices and uses a parameter prefetch pipeline (controlled by stage3_prefetch_bucket_size) to overlap data movement with computation.
Both approaches share a common architectural pattern: hierarchical memory tiering. Model weights reside in the tier with the most capacity (disk > CPU > GPU), and are promoted to GPU memory only when needed for the current computation step. The key performance characteristics are determined by the bandwidth of data movement between tiers and the degree of overlap between data transfer and computation.
Usage
Apply this principle when establishing performance baselines for offloaded LLM inference. These baselines represent the state of the art from general-purpose frameworks before specialized systems like FlexLLMGen optimize the offloading pipeline. Understanding these baselines is critical for quantifying the improvements that tensor-level scheduling and compute-I/O overlap provide.
Theoretical Basis
Memory Hierarchy Offloading
Large model inference faces a fundamental constraint: model memory footprint (parameters + KV cache + activations) often exceeds GPU memory. The approach of offloading parameters to lower tiers in the memory hierarchy introduces a latency penalty proportional to:
T_offload = model_bytes / bandwidth(tier_source, GPU)
For CPU offloading via PCIe 4.0 x16, the theoretical bandwidth is approximately 32 GB/s. For NVMe offloading, sequential read bandwidth is typically 3-7 GB/s depending on the device.
Prefill vs. Decode Decomposition
Autoregressive generation consists of two distinct phases with different computational profiles:
- Prefill phase: Processes all prompt tokens in a single forward pass. This is compute-bound for large batch sizes and exhibits high arithmetic intensity.
- Decode phase: Generates tokens one at a time, each requiring a full model forward pass over a single token position. This is memory-bandwidth-bound because the ratio of computation to data movement is very low.
The offloading overhead disproportionately affects the decode phase, since each of the gen_len - 1 decode steps requires loading the full model parameters from the offload tier.
Latency Projection
For very long generation sequences, measuring full decode latency becomes impractical. A common technique is to run a shorter generation (e.g., 8 tokens) and project the total latency based on the per-step decode cost, assuming that per-step latency is approximately constant after the first few tokens stabilize the pipeline.