Implementation:InternLM Lmdeploy Profiler

Knowledge Sources	InternLM_Lmdeploy
Domains	Benchmarking, Performance, Utilities
Last Updated	2026-02-07 15:00 GMT

Overview

A lightweight profiling framework consisting of Session and Profiler classes that track per-request timing data and compute aggregate performance metrics for lmdeploy benchmarks.

Description

The lmdeploy/profiler.py module provides the core profiling infrastructure used by all lmdeploy benchmark scripts. It consists of two classes:

Session class: Represents a single inference request and tracks its lifecycle:

• tick(n_token): Records the current timestamp and cumulative token count at each generation step.
• finish(status): Marks the session as SUCCESS, FAIL, or UNKNOWN.
• Stores input_len and req_output_len for validation.
• Maintains parallel lists ts (timestamps) and ns (token counts) for fine-grained timing analysis.

Profiler class: Manages a collection of sessions and computes aggregate metrics:

• new_session(input_len, output_len): Creates and registers a new Session.
• start() / finish(): Brackets the overall benchmark duration.
• compute_metrics(): Iterates over all successful sessions to compute:
  • TTFT (Time to First Token): ts[1] - ts[0]
  • TPOT (Time per Output Token): (ts[-1] - ts[1]) / (ns[-1] - ns[1])
  • E2E (End-to-end latency): ts[-1] - ts[0]
  • ITL (Inter-token latency): differences between consecutive timestamps
  • Throughput: total input/output tokens divided by elapsed time
  • RPS: successful requests divided by elapsed time
  • Percentile statistics (configurable, e.g., P50, P75, P95, P99)
• summarize(title, hyperparams): Prints a formatted ASCII table with all metrics.
• save_csv(csv_file, hyperparams): Appends metrics to a CSV file with configurable hyperparameter columns. Creates the header row if the file does not exist.

The module handles edge cases such as non-streaming output (falls back to total-time-based TPOT) and empty metric lists (substitutes infinity/zero).

Usage

Used by profile_pipeline_api.py, profile_throughput.py, and other benchmark scripts as the metrics collection and reporting layer.

Code Reference

Source Location

• Repository: InternLM_Lmdeploy
• File: lmdeploy/profiler.py
• Lines: 1-174

Signature

class Session:
    UNKNOWN = 0
    SUCCESS = 1
    FAIL = 2

    def __init__(self, input_len, req_output_len): ...
    def tick(self, n_token): ...
    def finish(self, status): ...

class Profiler:
    def __init__(self, stream_output: bool, percentages: List[int]): ...
    def new_session(self, *args, **kwargs) -> Session: ...
    def start(self): ...
    def finish(self): ...
    def compute_metrics(self): ...
    def summarize(self, title: str, hyperparams: List = None,
                  header=40, digits=10): ...
    def save_csv(self, csv_file: str, hyperparams): ...

Import

from lmdeploy.profiler import Profiler, Session

I/O Contract

Inputs

Name	Type	Required	Description
stream_output	bool	Yes (Profiler)	Whether streaming output is enabled (affects TTFT/ITL metrics)
percentages	List[int]	Yes (Profiler)	Percentile values to compute (e.g., [50, 75, 95, 99])
input_len	int	Yes (Session)	Number of input tokens for the request
req_output_len	int	Yes (Session)	Requested number of output tokens
n_token	int	Yes (tick)	Cumulative token count at this tick
status	int	Yes (finish)	Session status: SUCCESS (1) or FAIL (2)

Outputs

Name	Type	Description
output_throughput	float	Output tokens per second
input_throughput	float	Input tokens per second
rps	float	Requests per second
ttft_mean / ttft_stat	float / tuple	Time to first token statistics
tpot_mean / tpot_stat	float / tuple	Time per output token statistics
e2e_mean / e2e_stat	float / tuple	End-to-end latency statistics
itls_mean / itls_stat	float / tuple	Inter-token latency statistics
CSV file	file	Appended row with metrics (via save_csv)

Usage Examples

from lmdeploy.profiler import Profiler, Session

# Create a profiler tracking P50, P95, P99 percentiles
profiler = Profiler(stream_output=True, percentages=[50, 95, 99])

# Create sessions for requests
sess1 = profiler.new_session(input_len=128, req_output_len=256)
sess2 = profiler.new_session(input_len=64, req_output_len=128)

# Start benchmark timing
profiler.start()

# Simulate token generation for session 1
sess1.tick(0)    # First tick: marks start
sess1.tick(1)    # First token generated
sess1.tick(128)  # More tokens...
sess1.tick(256)  # All tokens generated
sess1.finish(Session.SUCCESS)

# Compute and display metrics
profiler.finish()
profiler.compute_metrics()
profiler.summarize(title='My Benchmark')

# Save to CSV
profiler.save_csv('results.csv', (('backend', 'turbomind'), ('tp', 1)))

Related Pages

• Environment:InternLM_Lmdeploy_CUDA_GPU_Runtime