Implementation:Vllm project Vllm Test Pipeline Config

Knowledge Sources	vllm
Domains	CI/CD, Testing, NVIDIA GPU
Last Updated	2026-02-08 00:00 GMT

Overview

Defines the main Buildkite test pipeline template for running the vLLM test suite on NVIDIA GPU hardware (default pipeline).

Description

This YAML file is the core testing infrastructure configuration for vLLM, specifying all test steps that run on NVIDIA GPUs (L4, A100, B200, H200). It is fed into a Jinja template at the vllm-project/buildkite-ci repository to generate the final pipeline YAML. The configuration covers fast-check tests, correctness tests, model tests, distributed multi-GPU tests, kernel tests, and more, with support for conditional execution based on source file changes.

Usage

Contributors interact with this file when adding new tests or modifying existing test configurations for NVIDIA hardware. The pipeline is triggered automatically by Buildkite on commits and pull requests. Steps can be configured with specific GPU types, GPU counts, parallelism levels, timeouts, and source file dependency filters to optimize CI resource usage.

Code Reference

Source Location

• Repository: vllm
• File: .buildkite/test-pipeline.yaml
• Lines: 1-1522

Signature

steps:
  - label: "Pytorch Nightly Dependency Override Check"
    mirror_hardwares: [amdexperimental, amdproduction, amdtentative]
    agent_pool: mi325_1
    grade: Blocking
    soft_fail: true
    source_file_dependencies: [...]
    commands: [...]

  - label: "Async Engine, Inputs, Utils, Worker Test"
    timeout_in_minutes: 15
    mirror_hardwares: [amdexperimental, amdproduction, amdtentative]
    source_file_dependencies: [...]
    commands: [...]

  - label: "Basic Correctness Test"
    fast_check: true
    torch_nightly: true
    gpu: a100
    num_gpus: 2
    commands: [...]

Import

# This file is consumed by the Jinja template engine in Buildkite CI:
# https://github.com/vllm-project/buildkite-ci/blob/main/scripts/test-template-aws.j2
#
# It is not imported directly; Buildkite processes this file to generate
# the final pipeline configuration for NVIDIA hardware tests.

I/O Contract

Inputs

Name	Type	Required	Description
label	string	Yes	The display name of the test step (emojis allowed)
commands	list[string]	Yes	Shell commands to execute for the test (incompatible with command)
command	string	Yes	Single shell command to execute (incompatible with commands)
timeout_in_minutes	int	No	Timeout for the step; uses default if not specified
gpu	string	No	GPU type selection (default: L4; supports a100, b200, h200)
num_gpus	int	No	Number of GPUs required (default: 1; supports 2, 4)
num_nodes	int	No	Simulate multi-node by launching multiple containers on one host
mirror_hardwares	list[string]	No	Additional hardware targets (e.g., [amdexperimental, amdproduction, amdtentative])
agent_pool	string	No	Specific agent pool for test execution
grade	string	No	Test grade classification (e.g., Blocking)
source_file_dependencies	list[string]	No	File path prefixes that trigger this test; if empty, test always runs
fast_check	bool	No	Whether to run on each commit in the fastcheck pipeline
fast_check_only	bool	No	Run only on the fastcheck pipeline
torch_nightly	bool	No	Whether to run against PyTorch nightly builds
optional	bool	No	Never run by default unless manually unblocked or scheduled nightly
soft_fail	bool	No	Allow step to fail without failing the entire pipeline
parallelism	int	No	Number of parallel shards using BUILDKITE_PARALLEL_JOB
working_dir	string	No	Directory for command execution (default: /vllm-workspace/tests)

Outputs

Name	Type	Description
Test results	Buildkite step results	Pass/fail status for each test step displayed in Buildkite UI
Test logs	stdout/stderr	Console output from pytest and other test runners

Usage Examples

# Adding a new test step for A100 GPUs with 2 GPUs:
- label: "My Distributed Test"
  gpu: a100
  num_gpus: 2
  timeout_in_minutes: 45
  source_file_dependencies:
    - vllm/distributed/
    - tests/distributed/
  commands:
    - pytest -v -s distributed/test_my_feature.py

# Adding a fast-check test:
- label: "Quick Validation Test"
  fast_check: true
  commands:
    - pytest -v -s test_quick_validation.py

Related Pages

• Environment:Vllm_project_Vllm_Buildkite
• Environment:Vllm_project_Vllm_NVIDIA_CUDA