Workflow:MaterializeInc Materialize Upgrade Testing
| Knowledge Sources | |
|---|---|
| Domains | Testing, Upgrade_Validation, Version_Management |
| Last Updated | 2026-02-08 21:00 GMT |
Overview
End-to-end process for validating that Materialize data and state survive version upgrades, restarts, and operational changes using the Check/Scenario testing framework.
Description
This workflow describes the upgrade testing framework built on the Check base class and Scenario orchestrator. Each Check defines a three-phase lifecycle: initialize (create objects and insert data), manipulate (modify data in two phases around upgrade events), and validate (verify data consistency). Scenarios compose multiple Checks with specific action sequences that simulate upgrade paths, restarts, and failure recovery. Over 80 Check implementations cover SQL types, CDC sources, sinks, privileges, clusters, and more. The framework supports testing upgrades from any previous release to the current version, zero-downtime deployment, and backup/restore recovery.
Usage
Execute this workflow when adding new Materialize features that must survive version upgrades, when validating a new release candidate against previous versions, or when implementing new Check classes for untested upgrade paths. The framework is run automatically in CI for every PR to verify backward compatibility.
Execution Steps
Step 1: Define Check Classes
Create a Check subclass in the checks/all_checks/ directory that implements the three lifecycle methods. The initialize() method creates database objects and inserts initial data. The manipulate() method returns a list of two actions that modify data before and after the upgrade event. The validate() method verifies that all data is consistent after the upgrade.
Key considerations:
- Each Check must be externally idempotent by default (validate can be called multiple times)
- Manipulate must return exactly two actions (one per phase, executed around the upgrade)
- Actions can be Testdrive scripts (SQL-based) or PyAction callables
- Set class attributes like enabled, externally_idempotent, and supports_forced_migrations as needed
Step 2: Select or Create Upgrade Scenarios
Choose from existing Scenarios or create new ones that define the sequence of actions around version transitions. Standard scenarios include UpgradeEntireMz (upgrade from last release), UpgradeEntireMzFromLatestSelfManaged (upgrade from latest self-managed version), and zero-downtime upgrade scenarios with rolling deploy generations.
Key considerations:
- Scenarios define action sequences: StartMz (old version) -> Initialize -> Manipulate(phase 1) -> KillMz -> StartMz (new version) -> Manipulate(phase 2) -> Validate
- Zero-downtime scenarios use deploy_generation to test rolling upgrades
- The base_version() method determines the starting version for the upgrade path
- Checks incompatible with a scenario are automatically filtered
Step 3: Configure Version Resolution
The framework resolves version information from published release lists. Helper functions fetch available minor versions, determine the last released version, and identify the previous self-managed version for multi-hop upgrade testing.
Key considerations:
- Version resolution uses the version_list module to query published releases
- Results are cached to avoid repeated network calls during test execution
- Upgrade paths can span multiple minor versions for thorough testing
- The current development version is treated as the upgrade target
Step 4: Execute Upgrade Scenarios
Run upgrade tests via mzcompose by invoking the platform-checks composition. The Scenario.run() method executes each action in sequence, managing service lifecycle, check execution, and implicit configuration. The executor coordinates between mzcompose services and the check phases.
Key considerations:
- StartMz actions configure the Materialize image version, system parameters, and deployment settings
- KillMz and restart actions simulate the upgrade boundary
- Implicit ConfigureMz steps handle RBAC setup and Kafka/Schema Registry connections
- Validation runs after all manipulations to verify data consistency
Step 5: Validate Results
The Validate action runs each Check's validate() method, comparing actual query results against expected values. Failures indicate upgrade regressions. Results are reported through the standard mzcompose test result mechanism and surfaced in CI.
Key considerations:
- Validation failures produce detailed error messages with expected vs actual values
- The framework supports running validation multiple times to check for non-determinism
- Failed checks are aggregated into the scenario's test results
- CI annotations link failures to relevant code changes