Principle:MaterializeInc Materialize Platform Check Pattern
| Knowledge Sources | |
|---|---|
| Domains | Testing, Upgrade_Validation |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
A state-transition testing pattern that validates database features survive version upgrades by executing initialize/manipulate/validate lifecycle phases across upgrade boundaries.
Description
The Platform Check Pattern is a testing methodology where each feature under test is encapsulated as a Check subclass with three lifecycle methods. The check's state transitions are interleaved with system upgrades to verify that:
- Data created before an upgrade remains accessible after
- Operations performed during upgrade transitions complete correctly
- Feature behavior is consistent across versions
This pattern extends the existing Check Lifecycle principle by providing a library of 93 concrete feature validations organized into categories (CDC sources, sinks, data types, SQL operations, access control, upsert, infrastructure, and more).
Usage
Use this pattern when adding new Materialize features that must be validated across version upgrades. Create a new Check subclass in checks/all_checks/ that tests the feature's critical paths.
Theoretical Basis
The pattern follows a state machine model:
# Abstract lifecycle (NOT implementation code)
State_0 = initialize() # Pre-upgrade: create objects, insert data
-> UPGRADE occurs
State_1 = manipulate()[0] # Post-upgrade: modify state
-> Optional additional UPGRADE
State_2 = manipulate()[1] # Further modifications
-> Final state
validate() # Assert expected state matches actual
The key invariant is: validate() must pass regardless of which upgrade path was taken between initialize() and validate().
State Transition Diagram
The lifecycle can be visualized as a linear state machine with upgrade boundaries:
+-------------+ +-------------------+ +------------------+ +------------------+ +------------+
| initialize()|---->| UPGRADE / BACKUP |---->| manipulate()[0] |---->| UPGRADE / BACKUP |---->|manipulate()|
| (phase 0) | | (boundary) | | (phase 1) | | (boundary) | | [1](ph. 2) |
+-------------+ +-------------------+ +------------------+ +------------------+ +-----+------+
|
v
+------------+
| validate() |
| (final) |
+------------+
The boundaries between phases are where upgrades, backup/restore operations, or Kubernetes StatefulSet replacements occur. The pattern guarantees that no matter what happens at these boundaries, the final validate() call should succeed.
Key Design Principles
Idempotence Classification
Checks are classified by their external idempotence -- whether the manipulate() actions can be safely re-run:
- Externally idempotent (default): The check can participate in all scenarios, including those that replay manipulate() phases (e.g., BackupAndRestoreToPreviousState)
- Not externally idempotent (decorated with @externally_idempotent(False)): The check uses external systems (Kafka, MySQL) where message ingestion or side effects cannot be safely replayed. These checks are excluded from replay-based scenarios.
Testdrive Encapsulation
Each lifecycle method returns Testdrive objects rather than executing SQL directly. This allows the scenario orchestrator to:
- Batch and order actions across multiple checks
- Interleave check actions with infrastructure operations (upgrades, backups)
- Execute actions through different executors (mzcompose, cloudtest)
Version-Conditional Logic
Checks can access self._base_version and self.current_version (both MzVersion) to conditionally adjust behavior based on the Materialize version being tested. This enables forward-compatible checks that gracefully handle features not yet available in older versions.
Scenario Composition
The pattern supports multiple scenario types that vary the placement and nature of the boundaries:
| Scenario Type | Boundary Operations | Purpose |
|---|---|---|
| Upgrade scenarios | Version upgrade between phases | Validate feature survival across version changes |
| BackupAndRestoreAfterManipulate | Backup/restore after all manipulations | Validate full state backup fidelity |
| BackupAndRestoreBeforeManipulate | Backup/restore mid-manipulation | Validate partial state restoration |
| BackupAndRestoreToPreviousState | Restore to earlier state, replay | Validate rollback correctness |
| BackupAndRestoreMulti | Repeated backup/restore at every phase | Validate repeated backup/restore stability |
| Cloudtest scenarios | K8s StatefulSet replacement | Validate upgrade in Kubernetes environments |