Principle:MaterializeInc Materialize Change Detection and Test Trimming
| Knowledge Sources | Build system optimization, dependency-aware testing, incremental CI strategies |
|---|---|
| Domains | Continuous Integration, Build Systems, Dependency Analysis, Test Optimization |
| Last Updated | 2026-02-08 |
Overview
Change detection and test trimming is the practice of only running CI test steps whose inputs have changed relative to a baseline branch, using dependency graph analysis and git diff to minimize CI cost and execution time.
Description
In large repositories, running the full test suite on every commit is prohibitively expensive. Incremental CI addresses this by analyzing which files have changed in a branch and determining which tests are affected by those changes. Only affected tests are executed; unaffected tests are trimmed from the pipeline.
This approach requires two key capabilities:
- Dependency Graph Construction: The system must know, for each test step, what its inputs are. Inputs can be:
- Explicit inputs: Glob patterns declared directly on the test step configuration (e.g.,
src/adapter/**). - Implicit inputs: Dependencies discovered automatically by analyzing which container images and Python modules a test composition uses. If a test runs via Docker Compose and uses images A and B, then the source files for A and B (and their transitive dependencies) are all inputs to that test.
- Explicit inputs: Glob patterns declared directly on the test step configuration (e.g.,
- Change Detection: Given the set of inputs for each test, the system compares the current branch against the baseline (typically
main) usinggit diffto determine which inputs have actually changed.
A test step is trimmed (skipped) if and only if:
- None of its inputs have changed, and
- No other untrimmed step depends on it (dependency propagation).
The dependency propagation rule ensures that if step B depends on step A, and step B's inputs have changed, then step A is also included in the pipeline even if its own inputs have not changed.
Usage
Apply change detection and test trimming when:
- The repository has a large number of independent test suites with well-defined input boundaries.
- Full CI runs are too slow or expensive for pull request builds.
- The build system can reliably enumerate the inputs for each test step.
- There is a stable baseline branch (e.g.,
main) to diff against.
Do not apply trimming when:
- Running on the main branch or tagged releases (all tests should run to ensure full coverage).
- Running with special build modes (coverage, sanitizers) where full test runs are expected.
- CI infrastructure glue code itself has changed, since changes to the CI system could affect any test.
Theoretical Basis
The theoretical foundation of test trimming is dependency graph analysis combined with change set computation.
Formal Model:
Let G = (V, E) be a directed acyclic graph where vertices V represent pipeline steps and edges E represent dependencies (A -> B means A depends on B). Each vertex v has an associated input set I(v), which is the union of:
- Explicit glob patterns declared on the step
- Source files for all container images used by the step (transitively resolved)
- Python modules imported by the test composition
Let D = git diff main...HEAD be the set of changed files. A step v is directly changed if I(v) intersect D != empty.
The set of needed steps is computed by:
changed = { v in V : I(v) intersect D != empty }needed = transitive_closure(changed, reverse(E))
That is, any step that is directly changed, plus any step that a changed step transitively depends on.
The pipeline is then restricted to only the steps in the needed set. Group structures and wait barriers are preserved only if they contain at least one needed step.
Python Import Analysis:
A subtlety is that test compositions are Python scripts that may import other Python modules. Changes to those imported modules should also trigger the test. The system uses static analysis (via a dedicated bin/ci-python-imports tool) to discover the transitive set of Python files imported by each composition, and includes those files in the input set.