Principle:Apache Dolphinscheduler Workflow Instance Recovery
| Knowledge Sources | |
|---|---|
| Domains | Distributed_Systems, Fault_Tolerance |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
A workflow recovery mechanism that restarts failed or suspended workflow instances from their last known state, re-executing only the incomplete tasks while preserving completed work.
Description
The Workflow Instance Recovery principle provides two recovery modes: recovery from failure (re-runs failed tasks and their downstream dependencies) and recovery from suspend (resumes suspended tasks). Both modes are exposed through the IWorkflowControlClient RPC interface and use dedicated trigger classes on the master side. Recovery preserves the workflow instance identity and only re-executes tasks that did not complete successfully.
Usage
Recovery can be triggered automatically during failover or manually by users through the API/UI. Use triggerFromFailureTasks for workflows that failed mid-execution and triggerFromSuspendTasks for workflows that were explicitly paused/suspended.
Theoretical Basis
Recovery follows a Checkpoint-Resume Pattern:
- Checkpoint: Task completion states persisted in database
- Resume: Only re-execute tasks with non-SUCCESS status
- Dependency preservation: Downstream tasks of failed tasks are also re-queued
recoverFromFailure(workflowInstanceId):
instance = loadWorkflowInstance(id)
failedTasks = findTasksWithStatus(FAILED, KILLED)
downstreamTasks = findAllDownstream(failedTasks)
requeue(failedTasks + downstreamTasks)
resume(instance)
recoverFromSuspend(workflowInstanceId):
instance = loadWorkflowInstance(id)
suspendedTasks = findTasksWithStatus(PAUSED, READY)
resume(suspendedTasks)