Implementation:Deepspeedai DeepSpeed InferenceEngine Profiling

Overview

Concrete tool for profiling inference forward pass latency provided by the DeepSpeed library.

Implementation Type

Method (instance methods of InferenceEngine)

Detailed Description

InferenceEngine.profile_model_time() enables profiling mode on the inference engine. InferenceEngine.model_times() returns the list of recorded forward pass times in milliseconds and clears the internal list for the next profiling interval.

profile_model_time() (lines L193-201):

This method enables profiling by:

1. Checking that profiling is not already enabled and CUDA graphs are not active (hook-based profiling is incompatible with engine-level CUDA graph replay).
2. Registering a forward pre-hook (_pre_forward_hook) on the underlying model that starts a timer before each forward pass.
3. Registering a forward post-hook (_post_forward_hook) on the underlying model that stops the timer after each forward pass and appends the elapsed time to the internal _model_times list.
4. Setting model_profile_enabled = True and storing the use_cuda_events flag.
5. If CUDA events are requested, creating a SynchronizedWallClockTimer instance for GPU-accurate timing.

Pre-hook timing (lines L230-235):

• CUDA events mode: Calls self.timers(INFERENCE_MODEL_TIMER).start() which inserts a CUDA event into the stream.
• Wall-clock mode: Calls get_accelerator().synchronize() then records time.time().

Post-hook timing (lines L237-245):

• CUDA events mode: Calls self.timers(INFERENCE_MODEL_TIMER).stop() and .elapsed(reset=True) to get the elapsed time in milliseconds.
• Wall-clock mode: Calls get_accelerator().synchronize(), records time.time(), and computes the difference in milliseconds.
• Appends the elapsed time to self._model_times.

model_times() (lines L525-534):

1. Asserts that profiling is enabled.
2. Captures the current _model_times list reference.
3. If CUDA graphs are enabled and the list is empty, raises a ValueError explaining that CUDA graph profiling for GPT-style models is not supported.
4. Resets self._model_times to an empty list.
5. Returns the captured list.

CUDA graph profiling fallback (lines L564-581 in forward()):

When both profiling and CUDA graphs are enabled, the forward() method itself handles timing instead of hooks:

1. Synchronize GPU and record wall-clock start time before graph replay.
2. After graph replay, synchronize GPU and compute elapsed time.
3. Append elapsed time to _model_times.

Code Reference

• Repository: https://github.com/deepspeedai/DeepSpeed
• File: deepspeed/inference/engine.py
• Lines: L193-201 (profile_model_time), L230-245 (hooks), L525-534 (model_times)
• Signatures:
  • def profile_model_time(self, use_cuda_events: bool = True) -> None
  • def model_times(self) -> list[float]
• Import: Accessed via the InferenceEngine returned by deepspeed.init_inference()

Parameters

Method	Parameter	Type	Required	Default	Description
profile_model_time	use_cuda_events	bool	No	True	Use CUDA event timing (GPU-accurate) instead of wall-clock timing

I/O

Direction	Method	Name	Type	Description
Input	profile_model_time	use_cuda_events	bool	Whether to use CUDA events for GPU-accurate timing
Output	model_times	times	list[float]	List of forward pass durations in milliseconds

Usage Example

import deepspeed
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Setup model and engine
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    torch_dtype=torch.float16
)
engine = deepspeed.init_inference(
    model,
    dtype=torch.float16,
    replace_with_kernel_inject=True
)

# Enable profiling with CUDA events (GPU-accurate timing)
engine.profile_model_time(use_cuda_events=True)

# Run inference over test data
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
test_prompts = [
    "DeepSpeed is",
    "Large language models are",
    "Inference optimization helps"
]

for prompt in test_prompts:
    tokens = tokenizer(prompt, return_tensors="pt").to("cuda")
    engine(**tokens)

# Retrieve and analyze timing results
times = engine.model_times()  # list of float, in milliseconds
avg_latency = sum(times) / len(times)
min_latency = min(times)
max_latency = max(times)

print(f"Forward passes: {len(times)}")
print(f"Average latency: {avg_latency:.2f} ms")
print(f"Min latency: {min_latency:.2f} ms")
print(f"Max latency: {max_latency:.2f} ms")

# model_times() clears the internal list, so subsequent calls
# will only return times from new forward passes
engine.profile_model_time(use_cuda_events=True)

Knowledge Sources

• https://github.com/deepspeedai/DeepSpeed
• https://www.deepspeed.ai/tutorials/inference-tutorial/
• https://developer.nvidia.com/blog/how-implement-performance-metrics-cuda-cc/

Relationships

Principle:Deepspeedai_DeepSpeed_Inference_Profiling

Metadata

• Workflow: Inference_Engine_Optimization
• Type: Implementation
• Last Updated: 2026-02-09 00:00 GMT