Implementation:MaterializeInc Materialize Mkpipeline Sh

Knowledge Sources	Materialize CI infrastructure, Buildkite pipeline upload API
Domains	Continuous Integration, Shell Scripting, Container Image Management
Last Updated	2026-02-08

Overview

Concrete Bash script for CI pipeline bootstrapping provided by Materialize's ci/mkpipeline.sh, which checks builder image availability across architectures and flavors before delegating to the Python pipeline generator.

Description

mkpipeline.sh is the entry point for Materialize's Buildkite CI pipeline. Its path is hardcoded in the Buildkite UI, making it the first script executed for every CI build. Because the project's Python toolchain is only available inside the CI builder image, this script is necessarily written in Bash.

The script performs the following operations:

Accepts the pipeline name as its first positional argument (defaulting to test).
Iterates over all architecture-flavor combinations (x86_64 and aarch64 crossed with stable, nightly, and min) in parallel background subshells.
Checks image existence by running bin/ci-builder exists $flavor with the MZ_DEV_CI_BUILDER_ARCH environment variable set to the target architecture.
Collects missing images by writing arch:flavor pairs to a temporary file.
If any images are missing, it builds a Buildkite pipeline YAML with bootstrap steps for each missing image, a wait barrier, and a final mkpipeline step that runs the Python pipeline generator inside the min builder image. This pipeline is uploaded via buildkite-agent pipeline upload.
If all images exist, it directly executes the Python pipeline generator via bin/ci-builder run min bin/pyactivate -m ci.mkpipeline.

Each bootstrap step uses a concurrency group keyed on the image tag (ci-builder:$tag) with concurrency of 1, preventing parallel builds of the same image. Architecture-specific agents are selected: builder-linux-x86_64 for x86_64 and builder-linux-aarch64-mem for aarch64.

Usage

This script is invoked automatically by Buildkite at the start of every CI build. It should not typically be called manually. The script accepts the pipeline name and any additional arguments, which are forwarded to the Python ci.mkpipeline module.

Code Reference

Source Location

ci/mkpipeline.sh, lines 20-77.

Signature

#!/usr/bin/env bash
# Usage: ci/mkpipeline.sh [pipeline] [extra-args...]
# Default pipeline: test

Import

This is an external Bash script, not a Python module. It is invoked directly by Buildkite:

ci/mkpipeline.sh test

I/O Contract

Inputs

Name	Type	Description
`pipeline`	Positional argument (string)	Name of the pipeline to generate (default: `test`). Corresponds to a subdirectory under `ci/` containing a `pipeline.template.yml`.
`$@`	Additional arguments	Forwarded to `ci.mkpipeline` (e.g., `--dry-run`, `--coverage`).
`MZ_DEV_CI_BUILDER_ARCH`	Environment variable	Set internally per iteration to `x86_64` or `aarch64`.

Outputs

Name	Type	Description
Buildkite pipeline upload	YAML via stdin to `buildkite-agent pipeline upload`	If bootstrap is needed, a pipeline with bootstrap steps, a wait barrier, and the mkpipeline step.
Direct execution	Process exec	If no bootstrap is needed, directly execs the Python pipeline generator.

Usage Examples

Checking image existence and bootstrapping (internal logic):

# The script iterates architectures and flavors in parallel:
for arch in x86_64 aarch64; do
  for flavor in stable nightly min; do
    (
      if ! MZ_DEV_CI_BUILDER_ARCH=$arch bin/ci-builder exists $flavor; then
        echo "$arch:$flavor" >> "$tmpfile"
      fi
    ) &
  done
done
wait

Bootstrap pipeline YAML generated when images are missing:

steps:
  - label: bootstrap stable x86_64
    command: bin/ci-builder exists stable || bin/ci-builder push stable
    concurrency: 1
    concurrency_group: "ci-builder:v1234-stable-x86_64"
    agents:
      queue: builder-linux-x86_64
  - wait
  - label: mkpipeline
    command: bin/ci-builder run min bin/pyactivate -m ci.mkpipeline test
    priority: 200
    agents:
      queue: hetzner-x86-64-4cpu-8gb-mkpipeline
    retry:
      automatic:
        - exit_status: -1
          signal_reason: none
          limit: 2
        - signal_reason: agent_stop
          limit: 2

Direct execution when all images exist:

exec bin/ci-builder run min bin/pyactivate -m ci.mkpipeline "$pipeline" "$@"

Related Pages

Implements Principle

Principle:MaterializeInc_Materialize_Pipeline_Bootstrap

Requires Environment

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment