Heuristic:Triton inference server Server Dynamic Batching Tuning
| Knowledge Sources | |
|---|---|
| Domains | Optimization, Inference_Serving |
| Last Updated | 2026-02-13 17:00 GMT |
Overview
Configuration methodology for Triton's dynamic batcher that can increase throughput by 3.7x (from ~73 to ~272 infer/sec) by combining individual requests into larger batches.
Description
Dynamic batching is the single most impactful optimization for Triton Inference Server. It combines individual inference requests arriving from different clients into a single larger batch for GPU execution, dramatically improving throughput without requiring any model changes. The dynamic batcher operates at the scheduler level, accumulating requests up to a configurable maximum batch size and optional delay window.
The tuning process follows a specific recommended sequence: start with default settings (dynamic_batching { }), benchmark, then adjust batch size and delay parameters to trade latency for throughput within the latency budget.
Usage
Use this heuristic whenever deploying a model that supports batching (max_batch_size >= 1) and you want to maximize throughput. Apply the recommended configuration process step by step, using Performance Analyzer to validate each change.
The Insight (Rule of Thumb)
- Action 1: Enable dynamic batching with default settings by adding dynamic_batching { } to config.pbtxt.
- Action 2: Benchmark with perf_analyzer at increasing concurrency to establish the throughput curve.
- Action 3: If latency is within budget, increase max_batch_size and set max_queue_delay_microseconds to a non-zero value (start with 100 microseconds).
- Action 4: Do not set preferred_batch_size for most models. Only use it for TensorRT models with multiple optimization profiles where specific batch sizes give significantly better performance.
- Trade-off: Higher batch sizes and longer delays increase throughput but also increase latency per request. Delayed batching waits up to the configured microseconds for more requests to arrive.
Reasoning
GPUs execute most efficiently when processing large, uniform batches. Individual requests leave the GPU underutilized. Dynamic batching fills this gap by transparently combining requests without client awareness.
Empirical evidence from Triton documentation (docs/user_guide/optimization.md):
| Configuration | Throughput | p95 Latency |
|---|---|---|
| No batching, concurrency 2 | ~73 infer/sec | ~34ms |
| Dynamic batching, concurrency 8 | ~272 infer/sec | ~36ms |
| Dynamic batching + 2 instances | ~289.6 infer/sec | ~36ms |
The dynamic batcher achieves a 3.7x throughput improvement with only a marginal latency increase.
Delayed batching mechanics (from docs/user_guide/batcher.md): When a maximum or preferred batch cannot be formed, the batcher delays up to max_queue_delay_microseconds. If a new request arrives during the delay that allows forming a full batch, the batch is sent immediately. If the delay expires, the partial batch is sent as-is. This mechanism ensures no request waits indefinitely.