Implementation:Ray project Ray Buildkite Core Pipeline
| Knowledge Sources | |
|---|---|
| Domains | CI, Testing, Core |
| Last Updated | 2026-02-13 16:00 GMT |
Overview
This file defines the Buildkite "core tests" pipeline group that builds and tests Ray's core functionality, including C++ components, Python core, dashboard, scheduling, autoscaler, and minimal installation tests.
Description
The .buildkite/core.rayci.yml pipeline is the largest and most critical test pipeline in the Ray CI system. It creates multiple Docker build images via Wanda (core build, GPU core build, minimal build) across multiple Python versions, then runs extensive test suites using test_in_docker with parallelism. The test suites cover Python core tests, Redis tests, memory pressure tests, out-of-disk tests, doc tests, dashboard tests, debug/ASAN/UBSAN/TSAN sanitizer tests, wheel tests, minimal installation tests, cgroup tests, C++ tests, Java worker tests, HA integration tests, runtime env container tests, Spark-on-Ray tests, and multi-GPU tests. All steps depend on forge and ray-core-build.
Usage
Developers modify this file when adding new core test suites, changing test parallelism, updating Python version matrices, adding new sanitizer configurations, or adjusting which tests run on premerge versus postmerge builds. It is also updated when changing Docker build image configurations or test dependency chains.
Code Reference
Source Location
- Repository: Ray
- File:
.buildkite/core.rayci.yml - Lines: 1-526
Signature
group: core tests
depends_on:
- forge
- ray-core-build
- ray-dashboard-build
steps:
# builds
- name: corebuild-multipy
label: "wanda: corebuild-py{{matrix}}"
wanda: ci/docker/core.build.wanda.yaml
tags: cibase
matrix:
- "3.10"
- "3.12"
env:
PYTHON: "{{matrix}}"
BASE_TYPE: "build"
BUILD_VARIANT: "build"
depends_on: oss-ci-base_build-multipy
Import
Configuration file, referenced by the Buildkite CI pipeline system. Loaded as part of the RayCI pipeline group mechanism and depends on the forge, ray-core-build, and ray-dashboard-build pipeline steps.
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
forge |
dependency | yes | The forge build environment providing Bazel and tooling |
ray-core-build |
dependency | yes | Core Ray binary build artifacts |
ray-dashboard-build |
dependency | yes | Dashboard build artifacts |
oss-ci-base_build-multipy |
dependency | yes | Base CI Docker images for multiple Python versions |
oss-ci-base_gpu-multipy |
dependency | yes | GPU-enabled base CI Docker images |
oss-ci-base_test-multipy |
dependency | yes | Test base images for minimal builds |
manylinux-x86_64 |
dependency | conditional | Required for wheel tests and HA integration tests |
Outputs
| Name | Type | Description |
|---|---|---|
corebuild-multipy |
Docker image | Core build images for Python 3.10 and 3.12 |
coregpubuild-multipy |
Docker image | GPU-enabled core build images for Python 3.10 |
minbuild-core |
Docker image | Minimal core build images for Python 3.10-3.14 |
| Test results | CI artifacts | Test results and logs from all core test suites |
Usage Examples
The pipeline defines both build and test steps. Build steps create Docker images, and test steps use those images to run tests:
# Core Python tests with parallelism
- label: ":ray: core: python tests"
tags:
- python
- dashboard
instance_type: large
parallelism: 4
commands:
- bazel run //ci/ray_ci:test_in_docker -- //python/ray/tests/... core
--workers "$${BUILDKITE_PARALLEL_JOB_COUNT}"
--worker-id "$${BUILDKITE_PARALLEL_JOB}"
--parallelism-per-worker 3
--python-version 3.10 --build-name corebuild-py3.10
--except-tags custom_setup
# Minimal installation tests across Python 3.10-3.14
- label: ":ray: core: minimal tests {{matrix}}"
tags:
- python
- min_build
commands:
- python ./ci/env/check_minimal_install.py
- bazel run //ci/ray_ci:test_in_docker -- //python/ray/tests/... core
--only-tags minimal
--python-version {{matrix}}
--build-name minbuild-core-py{{matrix}}
matrix: ["3.10", "3.11", "3.12", "3.13", "3.14"]
# C++ sanitizer tests (blocked on premerge)
- block: "run cpp sanitizer tests"
key: block-core-cpp-sanitizer-tests
- label: ":ray: core: cpp asan tests"
commands:
- RAYCI_DISABLE_TEST_DB=1 bazel run //ci/ray_ci:test_in_docker --
//:all //src/... core --build-type asan-clang
depends_on: block-core-cpp-sanitizer-tests