Implementation:MaterializeInc Materialize Pipeline Template Step
| Knowledge Sources | ci/mkpipeline.py, ci/test/pipeline.template.yml
|
|---|---|
| Domains | Continuous Integration, Pipeline Configuration, Buildkite |
| Last Updated | 2026-02-08 |
Overview
Concrete pattern for registering integration tests as Buildkite CI pipeline steps, provided by the pipeline template YAML format and the trim_tests_pipeline() function in ci/mkpipeline.py.
Description
Materialize's CI pipelines are defined as YAML template files (e.g., ci/test/pipeline.template.yml) that are processed by ci/mkpipeline.py to produce the final Buildkite pipeline. Each integration test is registered as a step in the template using the ./ci/plugins/mzcompose Buildkite plugin, which specifies the composition to run.
The trim_tests_pipeline() function in ci/mkpipeline.py processes the template to determine which steps need to run. It resolves mzbuild image dependencies, discovers Python module imports, combines these with explicit input globs, and uses git diff to identify changed inputs. Steps are trimmed if none of their inputs have changed and no dependent step needs them.
The template supports step grouping, parallelism, agent queue selection, timeout configuration, and inter-step dependencies via depends_on. Steps can be marked to skip coverage or sanitizer builds using the coverage and sanitizer fields.
Usage
Use this pattern when:
- Adding a new integration test to the CI pipeline.
- Configuring parallelism or timeout for an existing test step.
- Adding explicit input triggers so a test runs when specific files change.
- Understanding the structure of the Buildkite pipeline template.
Code Reference
Source Location
| Component | File | Lines |
|---|---|---|
| Pipeline template | ci/test/pipeline.template.yml |
Full file |
trim_tests_pipeline() |
ci/mkpipeline.py |
L658-819 |
Signature
trim_tests_pipeline():
def trim_tests_pipeline(
pipeline: Any,
coverage: bool,
sanitizer: Sanitizer,
lto: bool,
) -> None
Pipeline step YAML schema (mzcompose plugin):
- id: <unique-step-id>
label: "<display label>"
depends_on: <step-id or list of step-ids>
timeout_in_minutes: <int>
inputs: [<glob patterns>] # optional explicit inputs
parallelism: <int> # optional parallel job count
plugins:
- ./ci/plugins/mzcompose:
composition: <composition-name>
ci-builder: <builder-image> # optional, defaults to standard
agents:
queue: <agent-queue-name>
coverage: skip # optional: skip in coverage builds
sanitizer: skip # optional: skip in sanitizer builds
Import
# mkpipeline.py is invoked as a script, not typically imported
# The template YAML is loaded by mkpipeline.py's pipeline loading logic
I/O Contract
Inputs
| Parameter | Type | Description |
|---|---|---|
pipeline |
dict (parsed YAML) |
The full pipeline template loaded from YAML, containing a steps list
|
coverage |
bool |
Whether this is a coverage build; steps marked coverage: skip are excluded
|
sanitizer |
Sanitizer |
Sanitizer mode; steps marked sanitizer: skip are excluded
|
lto |
bool |
Whether link-time optimization is enabled; affects mzbuild profile selection |
composition (step field) |
str |
The name of the mzcompose composition directory (e.g., "testdrive", "cluster")
|
inputs (step field) |
list[str] |
Glob patterns for explicit file inputs; combined with auto-discovered dependencies |
depends_on (step field) |
list[str] | Other step IDs that this step depends on |
Outputs
| Output | Type | Description |
|---|---|---|
Modified pipeline |
dict |
The pipeline with unneeded steps removed; modified in place |
| Trimming report | stdout | For each step, prints whether it is included or excluded, along with its dependencies, globs, and images |
Usage Examples
Adding a new integration test to the pipeline template:
# In ci/test/pipeline.template.yml, add a step:
- id: my-new-test
label: "My New Test"
depends_on: build-aarch64
timeout_in_minutes: 30
plugins:
- ./ci/plugins/mzcompose:
composition: my-new-test
agents:
queue: hetzner-aarch64-4cpu-8gb
Adding explicit inputs to trigger on specific file changes:
- id: testdrive
label: "Testdrive"
depends_on: build-aarch64
timeout_in_minutes: 40
inputs: [test/testdrive]
parallelism: 20
plugins:
- ./ci/plugins/mzcompose:
composition: testdrive
agents:
queue: hetzner-aarch64-8cpu-16gb
Grouping related test steps:
- group: Builds
key: builds
steps:
- id: build-x86_64
label: ":rust: Build x86_64"
command: bin/ci-builder run stable bin/pyactivate -m ci.test.build
inputs:
- "*"
depends_on: []
timeout_in_minutes: 60
agents:
queue: l-builder-linux-x86_64
How trim_tests_pipeline() resolves dependencies (conceptual):
# For each step using the mzcompose plugin:
step = PipelineStep(config["id"])
# 1. Add explicit input globs
for inp in config.get("inputs", []):
step.extra_inputs.add(inp)
# 2. Add step dependencies
for d in config.get("depends_on", []):
step.step_dependencies.add(d)
# 3. Auto-discover image dependencies from the composition
for plugin in config["plugins"]:
if plugin_name == "./ci/plugins/mzcompose":
name = plugin_config["composition"]
composition = Composition(repo, name)
for dep in composition.dependencies:
step.image_dependencies.add(dep)
# Also add the mzcompose.py file and its imports
step.extra_inputs.add(composition_path)
for file in imported_files[composition_path]:
step.extra_inputs.add(file)
# 4. Check if any inputs changed via git diff
if have_paths_changed(step.inputs()):
changed.add(step.id)
# 5. Transitively include all steps depended upon by changed steps
for step_id in changed:
visit(steps[step_id]) # marks step and its dependencies as needed
# 6. Remove steps not in the "needed" set
pipeline["steps"] = [s for s in pipeline["steps"] if s["id"] in needed or ...]