Implementation:InternLM Lmdeploy Profile Throughput

Knowledge Sources	InternLM_Lmdeploy
Domains	Benchmarking, Performance, Throughput
Last Updated	2026-02-07 15:00 GMT

Overview

A low-level throughput benchmarking script that profiles lmdeploy engine performance by directly invoking the TurboMind or PyTorch engine's async inference API with concurrent sessions.

Description

The profile_throughput.py script measures raw engine throughput by bypassing the high-level pipeline API and interacting directly with the TurboMind or PyTorch engine instances. This provides the most accurate measurement of engine-level performance.

Engine class: Instantiates either TurboMind or PytorchEngine directly and creates model instances for concurrent inference. Each concurrent worker: 1. Pulls requests from a shared Queue 2. Calls async_stream_infer with ignore_eos=True to force generation of the requested output length 3. Optionally performs tokenization/detokenization 4. Tracks per-token timing via Session objects

Concurrency model: Uses asyncio.gather with multiple coroutines sharing a request queue. Each coroutine gets a unique session ID and increments by the concurrency count to avoid conflicts.

Unique features compared to profile_pipeline_api.py:

• Directly accesses engine internals (TurboMind.from_pretrained, create_instance)
• Supports --skip-tokenize and --skip-detokenize flags to measure pure engine throughput
• Supports --cancel-rate for testing request cancellation behavior
• Supports --use-uvloop for faster async event loop
• Supports data parallelism (--dp), context parallelism (--cp), and distributed executor backends
• Supports DLLM (Diffusion LLM) parameters for diffusion-based language models

Dataset sampling: Same ShareGPT and random sampling as profile_pipeline_api.py.

Usage

Used for measuring raw engine throughput. Requires a dataset file and model path.

Code Reference

Source Location

• Repository: InternLM_Lmdeploy
• File: benchmark/profile_throughput.py
• Lines: 1-436

Signature

class Engine:
    def __init__(self, model_path: str,
                 engine_config: Union[PytorchEngineConfig, TurbomindEngineConfig]): ...

    async def _inference(self, req_queue: Queue, session_id: int,
                         temperature: float, top_p: float, top_k: int,
                         stream_output: bool, skip_tokenize: bool,
                         skip_detokenize: bool, concurrency: int): ...

    def process_request(self, requests, profiler: Profiler,
                       concurrency, temperature, top_p, top_k,
                       stream_output, skip_tokenize, skip_detokenize,
                       cancel_rate): ...

def main(): ...

Import

from lmdeploy.messages import GenerationConfig, PytorchEngineConfig, TurbomindEngineConfig
from lmdeploy.profiler import Profiler, Session
from lmdeploy.tokenizer import DetokenizeState, Tokenizer

I/O Contract

Inputs

Name	Type	Required	Description
dataset	str	Yes	Path to the ShareGPT JSON dataset file
model_path	str	Yes	Path to the model or HuggingFace repo ID
--concurrency / -c	int	No	Number of concurrent inference sessions (default: 256)
--num-prompts / -n	int	No	Number of prompts to process (default: 5000)
--backend	str	No	Backend: turbomind or pytorch (default: turbomind)
--no-stream-output	flag	No	Disable streaming output
--skip-tokenize	flag	No	Pre-tokenize inputs before timing
--skip-detokenize	flag	No	Skip output detokenization
--cancel-rate	float	No	Probability of random request cancellation (default: 0)
--use-uvloop	flag	No	Use uvloop for async event loop
--tp	int	No	Tensor parallelism degree
--dp	int	No	Data parallelism degree
--cp	int	No	Context parallelism degree
--csv	str	No	Output CSV file path (default: ./profile_throughput.csv)

Outputs

Name	Type	Description
Console summary	text	Formatted metrics table with latency percentiles
CSV file	file	Benchmark results with hyperparameters

Usage Examples

# Profile TurboMind engine throughput
# python benchmark/profile_throughput.py \
#     /path/to/ShareGPT.json \
#     internlm/internlm2_5-7b-chat \
#     --backend turbomind \
#     --tp 1 \
#     --concurrency 128 \
#     --num-prompts 2000

# Profile with skip tokenize/detokenize for pure engine perf
# python benchmark/profile_throughput.py \
#     /path/to/ShareGPT.json \
#     Qwen/Qwen2.5-7B-Instruct \
#     --backend pytorch \
#     --tp 1 \
#     --skip-tokenize \
#     --skip-detokenize \
#     --concurrency 64

# Profile with random dataset and data parallelism
# python benchmark/profile_throughput.py \
#     /path/to/ShareGPT.json \
#     model_path \
#     --backend turbomind \
#     --dataset-name random \
#     --random-input-len 512 \
#     --random-output-len 256 \
#     --dp 2 --tp 4

Related Pages

• Environment:InternLM_Lmdeploy_CUDA_GPU_Runtime