Principle:Triton inference server Server Sequence Batching Testing

Overview

Sequence Batching Testing validates the correctness of Triton Inference Server's stateful sequence batching scheduler, which manages inference requests that belong to ordered sequences and must be routed to the same model instance to maintain internal state (e.g., hidden states in recurrent neural networks, attention KV caches in autoregressive transformers, or accumulated context in streaming audio models). Unlike dynamic batching where requests are independent, sequence batching introduces temporal dependencies between requests, making it one of the most complex and error-prone components in the inference server. This principle covers both functional correctness under normal operation and resilience under stress conditions.

Theoretical Basis

Statefulness in Inference Serving

Many production ML workloads are inherently stateful. A conversational AI model maintains dialogue context across turns. A video analytics model tracks objects across frames. A speech recognition model accumulates audio features across chunks. In all these cases, the server must:

Maintain affinity: Route all requests within a sequence to the same backend model instance so that the instance's internal state (GPU memory containing hidden states, KV cache entries, etc.) is available for the next step.
Preserve ordering: Execute requests within a sequence in strict temporal order. Out-of-order execution produces nonsensical results because each step depends on the state produced by the previous step.
Manage slot allocation: Sequence batcher "slots" are finite resources. Each slot represents one concurrent sequence that can be processed by a model instance. The batcher must allocate slots to new sequences, reclaim them when sequences end, and handle the case where all slots are occupied.

The Sequence Batcher Contract

Triton's sequence batcher uses control inputs -- special tensor inputs that signal sequence boundaries -- to manage state. The key control signals are:

START: Indicates the first request in a new sequence. The backend should initialize its internal state.
END: Indicates the last request in a sequence. The backend should finalize and release its state.
READY: Indicates that the slot contains a valid request (as opposed to an idle padding request in a batch).
CORRID (Correlation ID): A unique identifier for the sequence, used for routing and state lookup.

Testing must verify that these control signals are correctly generated by the batcher and correctly interpreted by the backend. A misassigned START signal causes the backend to reset state mid-sequence, destroying accumulated context. A missed END signal causes slot leaks, eventually exhausting all available slots and blocking new sequences.

Direct vs. Oldest Strategy

Triton supports two sequence scheduling strategies:

Direct: Each sequence is assigned to a specific slot at start time and remains there until completion. This provides deterministic routing but can lead to underutilization if sequences have varying lengths.
Oldest: The batcher dynamically assigns the oldest waiting sequence to the next available slot. This improves utilization but requires more complex state management and is more susceptible to ordering bugs.

Both strategies must be tested for correctness, and the testing must verify that switching between strategies does not corrupt state or drop sequences.

Stress Testing: Why Normal-Path Testing Is Insufficient

Sequence batching exhibits failure modes that only manifest under load:

Slot exhaustion: When all slots are occupied, new sequences must be queued. The queueing behavior, timeout handling, and error signaling must be verified under saturation.
Rapid start/end cycling: A burst of very short sequences (single-request sequences) stress-tests the slot allocation and reclamation path, exposing race conditions in slot lifecycle management.
Interleaved timeouts: When a sequence times out mid-execution, the batcher must cleanly release the slot and signal the backend to discard state, without affecting other sequences sharing the same batch.
Connection drops: If a client disconnects mid-sequence, the server must detect the orphaned sequence and reclaim its slot, even though no END signal was received.

Stress testing with high concurrency, variable sequence lengths, and injected failures is essential to validate these edge cases.

Failure Mode	Cause	Impact	Test Strategy
State corruption	Misrouted request to wrong slot	Wrong inference results	Verify CORRID routing under load
Slot leak	Missing END signal processing	New sequences blocked	Rapid start/end cycling
Ordering violation	Concurrent requests in same sequence	Nonsensical output	Multi-threaded sequence submission
Starvation	All slots held by long sequences	New sequence timeouts	Mixed short/long sequence loads

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment