Principle:Mlflow Mlflow CI Test Matrix Generation
| Knowledge Sources | |
|---|---|
| Domains | Continuous Integration, Test Infrastructure |
| Last Updated | 2026-02-13 20:00 GMT |
Overview
Dynamic generation of CI test matrices from a declarative compatibility configuration, with version filtering, change detection, and result visualization.
Description
Cross-version testing ensures that a project's integrations remain functional across the full range of supported dependency versions. Rather than maintaining static CI configurations that must be manually updated with every dependency release, this principle uses a declarative configuration file to dynamically generate the test matrix at CI time.
The Matrix Generator reads a YAML configuration that declares, for each integration flavor, the package name, minimum and maximum supported versions, unsupported version exclusions, per-version dependency overrides, Python version requirements, test commands, and optional settings like pre-test scripts and runner specifications. For each flavor and test category (models, autologging), it:
- Fetches all released versions of the package from the package index (PyPI), including release dates.
- Filters to versions within the declared minimum-maximum range, excluding unsupported versions and very recently uploaded releases (to avoid testing unstable new releases).
- Reduces to the latest micro version within each minor version to keep the matrix tractable.
- Optionally further reduces by testing every N-th version.
- Always includes the declared minimum version to ensure baseline compatibility.
- Optionally appends a "dev" version entry when a development installation command is configured.
- Infers the appropriate Python version for each test, either from explicit per-version overrides or by querying the package's own Python version requirements metadata.
- Assembles a complete matrix item with install commands, test commands, version info, and runner configuration.
The generator supports several filtering modes: testing only flavors affected by file changes in a pull request (using path-based heuristics to map changed files to flavors), testing only items that differ from a reference configuration (to detect YAML config updates), restricting to specific flavors or versions, excluding dev versions, and selecting only the latest version per group. A validation pass checks that all test files for each flavor are covered by the declared test commands, and that all version-conditioned requirements actually match at least one tested version.
The resulting matrix is split into chunks that respect CI platform limits on matrix size, grouped by flavor to keep related jobs together.
The Result Visualizer provides observability into cross-version test outcomes over time. It fetches scheduled workflow run data and job results from the CI platform API, parses job names to extract the flavor/version/category tuple, and pivots the results into a date-indexed table. Each cell shows the test outcome (success, failure, cancelled) linked to the specific job run, making it easy to spot regressions, flaky tests, and patterns correlated with specific versions or time periods.
Usage
This principle applies in CI pipelines for pull request validation (testing only affected flavors), scheduled nightly runs (testing the full matrix), and after compatibility configuration updates. The visualizer is used for monitoring test health trends and triaging cross-version failures.
Theoretical Basis
The matrix expansion algorithm:
For each flavor in configuration:
For each category (models, autologging) in flavor:
1. Fetch all released versions V from the package index
2. Filter V to [minimum, maximum] range, excluding unsupported sets
3. Remove versions uploaded within the last 24 hours
4. Reduce to latest micro per minor version
5. Apply test_every_n_versions sampling if configured
6. Ensure minimum version is always included
7. For each surviving version v:
a. Compose install command: base package + version-matched extra requirements
b. Determine Python version: explicit override > inferred from package metadata
c. Determine runner type: explicit override > default
d. Emit a MatrixItem(flavor, category, version, install, run, python, ...)
8. If dev installation configured:
a. Emit a dev MatrixItem with the dev install command
The change-detection filtering:
Given a set of changed file paths:
1. Apply a regex to extract flavor names from paths (mlflow/<flavor>/ or tests/<flavor>/)
2. If the matrix generator script itself changed, test all flavors
3. If tracing modules changed, also include the langchain flavor
4. Intersect the full matrix with only items matching changed flavors
The reference-diff filtering:
1. Expand both the current and reference configurations into full matrices
2. The test set is the set difference: current_matrix - reference_matrix
3. This captures any items added, changed, or affected by config modifications