Implementation:MaterializeInc Materialize Mkpipeline Main

Knowledge Sources	Materialize CI pipeline generator, Buildkite pipeline upload API, mzbuild system
Domains	Continuous Integration, Pipeline-as-Code, YAML Generation, Python CLI
Last Updated	2026-02-08

Overview

Concrete Python function for CI pipeline YAML generation provided by Materialize's ci/mkpipeline.py:main(), which loads a pipeline template, applies a series of transformations (trimming, prioritization, agent routing, retry configuration), and uploads the result to Buildkite.

Description

The main() function is the entry point for Materialize's Python-based pipeline generator. It is invoked via python -m ci.mkpipeline <pipeline> from within a CI builder container. The function orchestrates the complete pipeline generation workflow:

Argument Parsing: Accepts command-line arguments for --dry-run, --coverage, --sanitizer, --priority, and a positional pipeline name.
Template Loading: Reads ci/<pipeline>/pipeline.template.yml, substitutes $RUST_VERSION with the current Rust compiler version, and parses the YAML.
LTO Determination: Computes whether link-time optimization should be enabled based on environment variables (CI_LTO, BUILDKITE_TAG, BUILDKITE_PULL_REQUEST, CI_RELEASE_LTO_BUILD) and pipeline name.
Parallel Hash Computation: Spawns a background thread to compute dependency hashes for both architectures (aarch64, x86_64) using the mzbuild repository, used later for build trimming.
Test Selection: If CI_TEST_IDS or CI_TEST_SELECTION environment variables are set, trims the pipeline to only the specified tests.
Test Trimming: On non-main, non-tag branches without coverage/sanitizer, trims unchanged test steps. If CI glue code has changed, performs a dry-run trim instead.
Pipeline Transformations: Applies a sequence of transformations:
- truncate_skip_length -- truncates skip reason strings
- handle_sanitizer_skip -- skips steps incompatible with sanitizer builds
- increase_agents_timeouts -- adjusts timeouts for coverage/sanitizer builds
- prioritize_pipeline -- sets step priorities based on branch/tag/author
- switch_jobs_to_aws -- routes high-priority jobs to AWS agents
- permit_rerunning_successful_steps -- allows manual reruns
- set_retry_on_agent_lost -- configures automatic retries
- set_default_agents_queue -- sets default agent queue
- unparallelize -- adjusts parallelism settings
- set_parallelism_name -- sets parallelism labels
- check_depends_on -- validates step dependency references
- add_version_to_preflight_tests -- injects version info
- move_build_to_lto -- enables LTO for release builds
- trim_builds -- skips image builds when registry is up to date
- add_cargo_test_dependency -- adds cargo test step dependencies
- add_nightly_deploy_dependency -- adds deploy step dependencies
- remove_dependencies_on_prs -- removes certain dependencies for PRs
- remove_mz_specific_keys -- strips Materialize-specific metadata keys
Upload: Serializes the pipeline to YAML and uploads via buildkite-agent pipeline upload (or prints with --dry-run).

Usage

This function is called as the main entry point of the ci.mkpipeline module. It is invoked from ci/mkpipeline.sh inside a CI builder container, or can be run locally for testing with the --dry-run flag.

Code Reference

Source Location

ci/mkpipeline.py, lines 76-211.

Signature

def main() -> int:
    """
    mkpipeline creates a Buildkite pipeline based on a template file and uploads it
    so it is executed.
    """

Import

from ci.mkpipeline import main

I/O Contract

Inputs

Name	Type	Description
`--dry-run`	CLI flag (bool)	If set, prints the pipeline YAML without uploading it to Buildkite.
`--coverage`	CLI flag (bool)	If set, generates a coverage build pipeline. Disables test trimming.
`--sanitizer`	CLI option (`Sanitizer` enum)	Sanitizer mode. Defaults to `CI_SANITIZER` env var or `none`. Disables test trimming.
`--priority`	CLI option (int)	Base priority offset for all steps. Defaults to `CI_PRIORITY` env var or `0`.
`pipeline`	Positional argument (str)	Name of the pipeline (e.g., `test`, `nightly`, `release-qualification`). Maps to `ci/<pipeline>/pipeline.template.yml`.
`BUILDKITE_BRANCH`	Environment variable	Current branch name. Used to determine trimming behavior.
`BUILDKITE_TAG`	Environment variable	Current tag. Non-empty for release builds; disables trimming and enables LTO.
`BUILDKITE_PULL_REQUEST`	Environment variable	Pull request number if applicable. Affects LTO determination.
`CI_TEST_IDS`	Environment variable	Comma-separated test step IDs to select (optional).
`CI_TEST_SELECTION`	Environment variable	Comma-separated test step names to select (optional).

Outputs

Name	Type	Description
Return value	`int`	Exit code: `0` on success.
Pipeline upload	YAML via `buildkite-agent pipeline upload`	The fully transformed pipeline YAML, uploaded to Buildkite (unless `--dry-run`).
stdout	Text	The pipeline YAML is printed before upload for debugging.

Usage Examples

Running the pipeline generator from the command line:

# Inside a CI builder container:
bin/pyactivate -m ci.mkpipeline test

# With coverage mode:
bin/pyactivate -m ci.mkpipeline --coverage test

# Dry run (print pipeline without uploading):
bin/pyactivate -m ci.mkpipeline --dry-run test

# With a specific sanitizer:
bin/pyactivate -m ci.mkpipeline --sanitizer address test

# With elevated priority:
bin/pyactivate -m ci.mkpipeline --priority 10 test

Calling main() programmatically:

import sys
from ci.mkpipeline import main

sys.argv = ["mkpipeline", "--dry-run", "test"]
exit_code = main()

Template loading and variable substitution (internal logic):

from pathlib import Path
import yaml
from ci.deploy.deploy_util import rust_version

with open(Path("ci/test/pipeline.template.yml")) as f:
    raw = f.read()
raw = raw.replace("$RUST_VERSION", rust_version())
pipeline = yaml.safe_load(raw)

Related Pages

Implements Principle

Principle:MaterializeInc_Materialize_Pipeline_YAML_Generation

Requires Environment

Uses Heuristic

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment