Principle:MaterializeInc Materialize Pipeline YAML Generation
| Knowledge Sources | Declarative CI/CD patterns, template-based code generation, Buildkite pipeline specification |
|---|---|
| Domains | Continuous Integration, Pipeline-as-Code, Template Processing, YAML Generation |
| Last Updated | 2026-02-08 |
Overview
Pipeline YAML generation is the pattern of defining CI pipelines declaratively via templates, then programmatically transforming and uploading the rendered pipeline to the CI system at build time.
Description
Modern CI systems such as Buildkite, GitHub Actions, and GitLab CI allow pipelines to be defined as YAML files. However, static YAML files cannot express conditional logic, dynamic step generation, or build-time optimizations. The template-generate-upload pattern addresses this limitation by introducing a programmatic layer between the template and the CI system.
The pattern works as follows:
- Template Authoring: A YAML template file (
pipeline.template.yml) defines the full set of pipeline steps, including metadata like inputs, dependencies, and plugin configurations. The template may contain placeholder variables (e.g.,$RUST_VERSION) that are resolved at generation time. - Template Loading and Variable Substitution: A pipeline generator script reads the template, performs string substitutions for dynamic values (such as the current Rust compiler version), and parses the result into a structured data representation.
- Programmatic Transformation: The generator applies a series of transformations to the parsed pipeline. These may include:
- Test trimming: Removing steps whose inputs have not changed.
- Build trimming: Skipping image builds when registry images are up to date.
- Priority adjustment: Setting step priorities based on branch, tag, and author.
- Agent selection: Routing steps to appropriate agent queues.
- Retry policies: Adding automatic retry logic for transient failures.
- Parallelism management: Adjusting or removing step parallelism.
- LTO configuration: Moving build steps to link-time optimization mode for release builds.
- Upload: The final pipeline is serialized back to YAML and uploaded to the CI system via its API (e.g.,
buildkite-agent pipeline upload).
This approach provides the benefits of declarative pipeline definition (readability, version control, code review) while enabling the dynamic behavior needed for a sophisticated CI system.
Usage
Apply the template-generate-upload pattern when:
- The CI pipeline requires conditional logic that cannot be expressed in static YAML.
- Build-time information (compiler versions, image hashes, branch names) must be injected into the pipeline.
- Different branches or build modes require different subsets of the pipeline.
- Pipeline transformations (trimming, prioritization, retry logic) are too complex for CI system built-in conditionals.
- The team wants to maintain a single source of truth for the pipeline structure (the template) while allowing programmatic customization.
Theoretical Basis
The template-generate-upload pattern is an instance of template-based code generation, a well-established technique in software engineering. The key insight is the separation of concerns:
- The template captures the structure of the pipeline: what steps exist, what they do, and how they relate to each other.
- The generator captures the policy: under what conditions steps should be included, how they should be configured, and what optimizations should be applied.
This separation allows pipeline authors to reason about structure and policy independently. Changes to the set of tests are made in the template; changes to CI optimization logic are made in the generator.
The transformation pipeline itself follows the pipeline pattern (also known as pipes and filters): the parsed pipeline data structure flows through a series of independent transformation functions, each of which modifies it in place. This design makes it easy to add, remove, or reorder transformations.
The key transformations form a directed sequence:
- Test selection / trimming (remove unneeded steps)
- Build optimization (skip cached images)
- Configuration adjustments (priorities, agents, retries, timeouts)
- Structural validation (dependency checking)
- Serialization and upload
Each transformation assumes only that its predecessors have already run, enabling a clean sequential composition.