Principle:Apache Dolphinscheduler State Reconciliation
| Knowledge Sources | |
|---|---|
| Domains | Distributed_Systems, Fault_Tolerance |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
A state reconciliation process that scans for orphaned workflows during master startup, cleans up stale failover markers, and sends alert notifications for recovery events.
Description
The State Reconciliation principle provides a safety net for the failover system. When a master server starts up, the globalMasterFailover method scans all workflow instances in non-finished states whose assigned master host is no longer active. For each orphaned instance, it initiates failover processing. Additionally, cleanHistoryFailoverFinishedMarks removes completed failover markers from the registry to prevent accumulation. Alert notifications via IAlertOperator keep administrators informed of failover events.
This reconciliation ensures that no workflow is permanently lost, even if the normal event-driven failover mechanism missed a failure (e.g., due to all masters being down simultaneously).
Usage
Automatically executed during master server startup. Also serves as a periodic safety check to catch any missed failover events.
Theoretical Basis
The reconciliation follows a Consistency Check Pattern:
globalMasterFailover():
activeMasters = clusterManager.getActiveMasters()
allRunningWorkflows = dao.findByStatusIn(RUNNING, READY, DELAY)
for workflow in allRunningWorkflows:
if workflow.masterHost not in activeMasters:
failoverWorkflow(workflow) // orphaned workflow
cleanHistoryFailoverFinishedMarks()
sendAlertNotification(failoverSummary)