Workflow:Apache Dolphinscheduler Workflow Execution Lifecycle
| Knowledge Sources | |
|---|---|
| Domains | Workflow_Orchestration, Task_Scheduling, Distributed_Systems |
| Last Updated | 2026-02-10 10:00 GMT |
Overview
End-to-end process for defining, triggering, and executing a DAG-based workflow in Apache DolphinScheduler, covering the complete lifecycle from definition to task completion.
Description
This workflow describes the complete lifecycle of a DolphinScheduler workflow instance, from initial definition through execution and completion. DolphinScheduler uses a DAG (Directed Acyclic Graph) model where workflows contain tasks connected by dependency relations. The Master server orchestrates execution by parsing the DAG, dispatching tasks to Workers based on load balancing strategies, and tracking state transitions. The process supports manual triggers, scheduled triggers, backfill triggers, and various execution modes including parallel and serial processing.
Usage
Execute this workflow when you need to create and run a data processing pipeline consisting of multiple dependent tasks. This is the primary use case of DolphinScheduler, applicable to ETL pipelines, data warehouse loading, machine learning training pipelines, report generation, and any multi-step automated process.
Execution Steps
Step 1: Define Workflow and Tasks
Create a workflow definition specifying the tasks and their dependency relations. Each task has a type (Shell, SQL, Spark, Sub-Workflow, etc.), execution parameters, and resource requirements. Task relations define the DAG structure by specifying pre-task and post-task connections. The definition is stored in the database with versioning support.
Key considerations:
- Workflow definitions support versioning for change tracking
- Task relations form a DAG (cycles are validated and rejected)
- Tasks can specify worker groups for targeted execution
- Environment variables and startup parameters can be configured per workflow
- Sub-workflow tasks enable modular, reusable pipeline components
Step 2: Trigger Workflow Execution
Initiate workflow execution through one of the supported trigger mechanisms. The trigger creates a command that the Master server picks up for processing. Each trigger type carries specific context (e.g., manual trigger includes user ID, schedule trigger includes cron context, backfill trigger includes time range).
Trigger types:
- Manual trigger via IWorkflowControlClient.manualTriggerWorkflow
- Schedule trigger via cron-based scheduler
- Backfill trigger for historical data processing
- Repeat trigger to re-run an existing instance
Step 3: Master Processes Command
The Master server's WorkflowEngine receives the trigger command and creates a workflow instance. It parses the DAG to determine task execution order, resolves dependencies, and identifies which tasks are ready to run (all predecessors completed). The workflow instance tracks overall state (Running, Success, Failure, Paused, Stopped).
Key considerations:
- DAG parsing identifies parallel execution opportunities
- Task dependency types: SUCCESS, FAILURE, or both
- Failure strategy determines behavior: END (fail workflow) or CONTINUE (run remaining tasks)
- Priority levels control execution order when resources are constrained
Step 4: Dispatch Tasks to Workers
Ready tasks are dispatched to Worker nodes through the RPC framework. The WorkerGroupDispatcherCoordinator selects a target Worker based on the configured load balancing strategy (Random, RoundRobin, FixedWeightedRoundRobin, DynamicWeightedRoundRobin). The task dispatch includes all execution parameters, environment settings, and resource references.
Key considerations:
- Worker group assignment controls which Workers can execute a task
- Load balancing distributes work across available Workers
- Slot-based distribution ensures fair resource allocation in multi-Master deployments
- Task groups can limit concurrent execution of specific task types
Step 5: Worker Executes Task
The Worker's PhysicalTaskExecutor receives the dispatched task and executes it in its target environment. Task execution produces logs, output parameters, and a final status. The Worker reports task progress and completion status back to the Master via RPC using ITaskExecutorEventListener.
Key considerations:
- Different task types have different executors (Shell, SQL, Spark, etc.)
- Task execution is isolated with configurable timeouts
- Output parameters can be passed to downstream tasks
- Log collection enables post-execution debugging
Step 6: Track Completion and Update State
As tasks complete, the Master updates the workflow DAG state, releases completed task slots, and identifies newly ready downstream tasks for dispatch. When all tasks have completed (or the failure strategy triggers termination), the workflow instance transitions to its final state. Execution results, timing, and logs are persisted for auditing.
Key considerations:
- Workflow state transitions: SUBMITTED_SUCCESS -> RUNNING_EXECUTION -> SUCCESS/FAILURE/STOP/PAUSE
- Completed workflows can be re-triggered or recovered from failure points
- Execution history is retained for monitoring and analysis
- Alert notifications can be configured for workflow completion or failure