Principle:MaterializeInc Materialize Randomized Chaos Testing
| Knowledge Sources | |
|---|---|
| Domains | Testing, Chaos Engineering, Resilience |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Randomized Chaos Testing is the principle of validating system correctness by executing randomly selected sequences of normal operations and disruptive events, ensuring Materialize maintains data consistency and recovers gracefully from failures.
Motivation
Deterministic tests exercise known failure paths but cannot discover unexpected interactions between concurrent operations and failures. Production systems experience unpredictable combinations of process crashes, restarts, network partitions, and workload patterns. Randomized chaos testing systematically explores this combinatorial space by generating thousands of action sequences that interleave creation, ingestion, validation, and disruption.
Key Tenets
Capability-Based Action Selection
Actions declare their preconditions (required capabilities) and postconditions (provided capabilities). The framework only selects actions whose requirements are currently satisfied, ensuring valid test sequences without manual state tracking. For example, CreateSource requires a Kafka topic capability, while ValidateView requires an existing view capability.
Weighted Random Exploration
Each scenario assigns numeric weights to actions controlling their selection probability. High-weight actions (e.g., Ingest: 100, DML: 30) execute frequently to build up state, while disruptive actions (e.g., MzStop: 1, KillClusterd: 5) execute less often but inject realistic failure conditions.
Validation After Disruption
Every scenario includes finalization steps that restart all services and run ValidateAll to verify data consistency after the chaos sequence completes. Some scenarios also include BackupAndRestore validation to ensure durability across backup cycles.
Composable Scenarios
Scenarios are composable through inheritance. The base Scenario class defines common bootstrap (Kafka, CockroachDB, blob store, Materialize, storaged, balancerd) and finalization sequences. Specialized scenarios add their own data sources and action weights while inheriting the base infrastructure setup.
Constraints
- Some action combinations are known to be incompatible (e.g., separate clusterd not supported by
Mz0dtDeploy) - PostgreSQL CDC sources cannot be backed up and restored due to LSN reconfirmation constraints
- The maximum number of concurrent objects (tables, views, sources) is bounded by
max_objectsparameters inActionFactoryinstances - Test execution is bounded by both action count and wall-clock time (
max_execution_time)