Implementation:Triton inference server Server L0 Cuda Graph Test
| Knowledge Sources | |
|---|---|
| Domains | Testing, CUDA_Optimization |
| Last Updated | 2026-02-13 17:00 GMT |
Overview
QA test script for validating CUDA graph optimization in the Triton Inference Server.
Description
This test validates that CUDA graph capture and execution work correctly within the Triton Inference Server. CUDA graphs reduce kernel launch overhead by recording a sequence of GPU operations and replaying them with minimal CPU involvement. The script configures models with CUDA graph optimization enabled, verifies that graph capture succeeds during warmup, and confirms that inference results remain numerically correct when executed via captured graphs. It tests various graph capture scenarios including different input shapes, batch sizes, and model types. The test also validates that the server gracefully handles cases where graph capture is not possible and falls back to regular execution.
Usage
Run as part of the Triton QA test suite to validate CUDA graph optimization. Typically executed in CI/CD pipelines within the Triton Docker container environment.
Code Reference
Source Location
- Repository: Triton Inference Server
- File: qa/L0_cuda_graph/test.sh
- Lines: 1-344
Signature
#!/bin/bash
# Key functions and variables
source ../common/util.sh
SERVER=/opt/tritonserver/bin/tritonserver
SERVER_ARGS="--model-repository=${MODELDIR}"
SERVER_LOG="./cuda_graph_test.server.log"
CLIENT_LOG="./cuda_graph_test.client.log"
Import
# Sources common utilities
source ../common/util.sh
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| DATADIR | env var | No | Directory containing test data |
| TRITON_DIR | env var | No | Path to Triton server installation |
| MODEL_REPO | directory | Yes | Generated test model repository with CUDA graph-enabled models |
Outputs
| Name | Type | Description |
|---|---|---|
| exit code | int | 0 on success, 1 on failure |
| test logs | files | Server logs and test output in test directory |
Usage Examples
Running the Test
cd qa/L0_cuda_graph/
bash test.sh