Principle:Deepspeedai DeepSpeed Inference Profiling

Overview

Measuring inference latency per forward pass using synchronized timing to evaluate DeepSpeed inference optimization effectiveness.

Detailed Description

Inference profiling enables measuring the actual latency of individual forward passes through the optimized model. This is essential for benchmarking different optimization configurations and understanding the real-world performance impact of DeepSpeed's inference engine.

Why profiling matters:

Different inference optimization configurations yield different latency/throughput tradeoffs. Profiling allows systematic comparison of:

Kernel injection vs. standard PyTorch execution — quantifying the speedup from fused CUDA kernels.
Varying tensor parallelism sizes — understanding the communication overhead vs. compute reduction tradeoff as tp_size increases.
CUDA graphs enabled vs. disabled — measuring the CPU overhead reduction from graph replay.
Different precision modes (fp16 vs. bf16 vs. int8) — evaluating throughput gains from reduced precision.
Different batch sizes and sequence lengths — profiling the latency curve across workload dimensions.

Profiling modes:

DeepSpeed provides two timing mechanisms:

CUDA event timing (default, recommended): Uses SynchronizedWallClockTimer which inserts CUDA events into the GPU stream before and after the forward pass. CUDA events are synchronized with the GPU execution timeline, providing accurate GPU kernel execution time without being affected by CPU overhead or asynchronous execution. This is the most reliable method for measuring actual GPU computation time.

Wall-clock timing: Uses Python's time.time() with explicit get_accelerator().synchronize() calls before start and after end. This measures end-to-end latency including any CPU overhead, memory transfers, and synchronization. Useful for measuring the total time a user would observe but less precise for isolating GPU computation.

Profiling lifecycle:

Call profile_model_time() to enable profiling. This registers forward pre-hooks and post-hooks on the model.
Execute forward passes (either via forward() or generate()). Each pass records its latency.
Call model_times() to retrieve the list of recorded times (in milliseconds) and clear the internal list.
Compute statistics (average, p50, p95, p99) from the returned times.

Special handling for CUDA graphs: When CUDA graphs are enabled, the hook-based profiling mechanism is bypassed because hooks cannot be triggered during graph replay. Instead, the forward() method itself records wall-clock timing with GPU synchronization when both profiling and CUDA graphs are enabled.

Theoretical Basis

GPU timing with CUDA events works by inserting marker events into the GPU command stream. The GPU records timestamps when it reaches these events during execution. Because the events are in the GPU's own execution stream, the measured duration reflects actual GPU kernel execution time, unaffected by:

CPU-GPU asynchrony: CUDA operations are dispatched asynchronously from the CPU. Wall-clock timing on the CPU side may not reflect when the GPU actually starts and finishes work.
Kernel launch overhead: The CPU may return from a kernel launch call before the GPU begins executing the kernel.
Concurrent operations: Multiple CUDA streams or CPU threads may interleave, making CPU-side timing unreliable.

Wall-clock timing with explicit synchronization (torch.cuda.synchronize()) forces the CPU to wait until all GPU operations complete before recording the timestamp. This provides accurate end-to-end latency but includes:

Synchronization overhead itself (blocking the CPU until GPU finishes)
Any CPU-side overhead between the sync points (memory allocation, Python overhead)

For inference latency benchmarking, CUDA event timing is preferred because it isolates GPU computation time from framework overhead, providing a more stable and reproducible measurement.

Knowledge Sources

Relationships

Implementation:Deepspeedai_DeepSpeed_InferenceEngine_Profiling

Metadata

Workflow: Inference_Engine_Optimization
Type: Principle
Last Updated: 2026-02-09 00:00 GMT

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment