Environment:Ray project Ray CI Build Matrix Environment
| Knowledge Sources | |
|---|---|
| Domains | Infrastructure, CI_CD |
| Last Updated | 2026-02-13 16:35 GMT |
Overview
Buildkite CI/CD environment with a multi-Python (3.10-3.14), multi-architecture (x86_64/aarch64), and multi-platform (Linux/macOS/Windows) test matrix for Ray builds and releases.
Description
This environment defines the CI/CD infrastructure for the Ray project, hosted on Buildkite. The pipeline configuration spans multiple YAML files under `.buildkite/`, defining build and test matrices across Python versions (3.10, 3.11, 3.12, 3.13, 3.14), architectures (x86_64, aarch64/ARM64), and platforms (Linux, macOS, Windows). GPU tests run with CUDA 12.8 support. The CI uses conditional test rules (`.buildkite/test.rules.txt`) to determine which tests to run based on changed files, optimizing pipeline duration.
Usage
This environment is required for running the full CI test suite, building release wheels across all supported platforms, and validating pull requests. It underpins the Build and Release Pipeline workflow and ensures cross-platform compatibility.
System Requirements
| Category | Requirement | Notes |
|---|---|---|
| CI Platform | Buildkite | Self-hosted runners |
| Python | 3.10, 3.11, 3.12, 3.13, 3.14 | Full matrix |
| Architecture | x86_64, aarch64 | ARM64 via builder-arm64 instances |
| GPU | CUDA 12.8 | For GPU test suite |
| OS (Linux) | Ubuntu-based Docker containers | Primary CI platform |
| OS (macOS) | macOS runners | Wheel builds and tests |
| OS (Windows) | Windows runners | Wheel builds and limited tests |
Dependencies
CI Pipeline Files
- `.buildkite/base.rayci.yml` (Python version matrix, base images)
- `.buildkite/build.rayci.yml` (build pipeline)
- `.buildkite/core.rayci.yml` (core tests)
- `.buildkite/serve.rayci.yml` (Serve tests)
- `.buildkite/rllib.rayci.yml` (RLlib tests)
- `.buildkite/data.rayci.yml` (Data tests)
- `.buildkite/_wheel-build.rayci.yml` (wheel build pipeline)
- `.buildkite/test.rules.txt` (conditional test rules)
Build Tools
- `bazel` = 6.5.0
- `maven` >= 3.x (for Java builds)
- `node` (version 14 on macOS, 22.4.1 on Windows)
- `setuptools` = 80.9.0 (for macOS wheel builds)
Wheel Build Platforms
- Linux: manylinux2014 (glibc 2.17 compatible)
- macOS: 10.9+ compatible, ARM64 and x86_64
- Windows: Visual Studio toolchain, Node.js 22.4.1
Credentials
CI runners require:
- `BUILDKITE_AGENT_TOKEN`: Buildkite agent authentication
- AWS credentials for artifact storage (via `.buildkite/copy_files.py`)
- Docker registry credentials for image publishing
Quick Install
# CI environments are pre-configured on Buildkite runners.
# For local reproduction:
pip install -r ci/repro-ci-requirements.txt
python ci/repro-ci.py # Reproduces CI environment locally
Code Evidence
Python version matrix from `.buildkite/base.rayci.yml:6-11`:
matrix:
- "3.10"
- "3.11"
- "3.12"
- "3.13"
- "3.14"
Wheel build matrix from `.buildkite/_wheel-build.rayci.yml:7-12`:
matrix:
- "3.10"
- "3.11"
- "3.12"
- "3.13"
- "3.14"
aarch64 constraint from `ci/ci.sh:20`:
# Resolution currently does not work on aarch64
Node.js version for Windows wheel builds from `python/build-wheel-windows.sh:58`:
choco install nodejs --version=22.4.1 -y
Common Errors
| Error Message | Cause | Solution |
|---|---|---|
| `Resolution currently does not work on aarch64` | Dependency resolution fails on ARM64 | Known CI limitation; use x86_64 for dependency resolution |
| Wheel build fails on macOS ARM64 | Architecture mismatch | Ensure `uname -m` returns arm64 and correct Miniforge is used |
| Test rules not matching | Changed files not covered by test.rules.txt | Check `.buildkite/test.rules.txt` for file pattern coverage |
Compatibility Notes
- aarch64: Dependency resolution has known issues on ARM64. The CI works around this by performing resolution on x86_64 and running pre-resolved builds on ARM64.
- Windows: Limited test coverage compared to Linux. Node.js version differs from macOS (22.4.1 vs 14).
- macOS: Builds both x86_64 and ARM64 wheels. Uses setuptools 80.9.0 pinned version.
- GPU tests: Run on dedicated GPU instances with CUDA 12.8 support.