Principle:MaterializeInc Materialize Upgrade Scenario Execution

Knowledge Sources	misc/python/materialize/checks/scenarios.py, misc/python/materialize/checks/mzcompose_actions.py
Domains	Upgrade Testing, Orchestration, Stateful Execution
Last Updated	2026-02-08

Overview

Stateful upgrade execution orchestrates start/stop cycles of database instances at different versions while maintaining state continuity through interleaved check phases.

Description

The Upgrade Scenario Execution principle addresses the problem of orchestrating a complex sequence of operations that span multiple Materialize versions. Unlike unit tests that operate in a single process, upgrade tests must:

1. Start a Materialize instance at a specific version (possibly from a Docker image tagged with that version).
2. Run check phases (initialize, manipulate) against the running instance.
3. Kill the instance, preserving its on-disk state (catalog, blob store, metadata store).
4. Start a new instance at a different version, using the same persistent state.
5. Run further check phases (manipulate, validate) against the upgraded instance.
6. Repeat for multi-hop upgrades.

The execution engine must guarantee that:

• State continuity is maintained across start/stop boundaries. The metadata store (Cockroach), blob store, and external services persist across instance lifecycles.
• Version identity is verified after each start. The executor confirms that the running Materialize version matches the expected tag.
• Implicit configuration is applied after each start. The run() method automatically inserts ConfigureMz actions after StartMz to set system limits and permissions.
• Action ordering is strictly sequential. Each action's execute() and join() methods are called in order, ensuring that no action begins before the previous one completes.

The execution model is designed to be deterministic: given the same scenario, check set, and seed, the exact same sequence of operations is performed. This makes failures reproducible.

Usage

Apply this principle when:

• Understanding how upgrade tests are orchestrated — the Scenario.run() method is the single entry point.
• Debugging upgrade test failures — the execution is a linear sequence of actions, each of which can be inspected independently.
• Adding new orchestration actions — new MzcomposeAction subclasses can be created and inserted into scenario action lists.
• Supporting new deployment modes — the StartMz action supports deploy generations, healthchecks, forced migrations, and read-only starts for zero-downtime upgrades.

Theoretical Basis

Upgrade scenario execution implements the Command pattern from object-oriented design: each action is an object that encapsulates a single operation (start, stop, configure, initialize, manipulate, validate). The scenario's run() method is the invoker that iterates over the command list.

The execution model can be described as a linear state machine where:

• The state is the set of running services and the accumulated persistent data.
• Each transition is an action that modifies the running service topology or the persistent data.
• The invariant is that after every Validate action, the check assertions hold.

State transitions for UpgradeEntireMz:

  [no services] --StartMz(v_old)--> [mz@v_old]
  [mz@v_old]    --Initialize------> [mz@v_old + initial state]
  [mz@v_old]    --Manipulate(1)---> [mz@v_old + manipulated state]
  [mz@v_old]    --KillMz----------> [no services, state on disk]
  [no services] --StartMz(v_new)--> [mz@v_new, state migrated]
  [mz@v_new]    --Manipulate(2)---> [mz@v_new + further mutations]
  [mz@v_new]    --Validate--------> [assertions pass]

The implicit insertion of ConfigureMz after StartMz actions implements the Template Method pattern: the base run() algorithm enriches the user-defined action list with required setup steps.

Related Pages

Implemented By

• Implementation:MaterializeInc_Materialize_Scenario_Run