Workflow:Astronomer Astronomer cosmos Watcher execution mode
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, dbt, Airflow, Orchestration, Performance |
| Last Updated | 2026-02-07 17:00 GMT |
Overview
End-to-end process for running dbt projects using the Watcher execution mode, a producer-consumer pattern that achieves up to 5x speedup over per-model execution by running a single dbt build while individual Airflow tasks monitor their corresponding model completion.
Description
This workflow covers Cosmos's Watcher execution mode (ExecutionMode.WATCHER), introduced in v1.11. Instead of running a separate dbt command per model (the default in local mode), Watcher mode runs a single dbt build command (the producer) while multiple consumer sensor tasks monitor XCom for the completion status of their individual models. This eliminates the overhead of repeated dbt startup, profile resolution, and partial parsing per model.
The producer task (DbtProducerWatcherOperator) executes the full dbt build and publishes each model's completion status to XCom in real-time via a background thread. Consumer tasks (DbtConsumerWatcherSensor) wait for their specific model to complete, then report success or failure. The consumer sensors support both deferrable (async) and synchronous polling modes.
This mode preserves the Airflow DAG's model-level visibility while dramatically reducing total execution time, making it suitable for large dbt projects where per-model execution creates excessive overhead.
Usage
Execute this workflow when you have a dbt project with many models and the per-model execution overhead is significant. Watcher mode is particularly effective when dbt startup time (loading profiles, partial parsing, dependency resolution) dominates the actual model computation time. It provides the best of both worlds: a single efficient dbt execution with per-model Airflow task visibility, logging, and retry capability.
Execution Steps
Step 1: Configure project and profile
Define the ProjectConfig with the dbt project path and ProfileConfig with a profile mapping, exactly as you would for local execution mode. The Watcher mode reuses the same project and profile configuration infrastructure. No changes to the dbt project itself are required.
Key considerations:
- Project path must be accessible on the Airflow worker at runtime
- Profile mapping works identically to local execution mode
- The dbt project should be compatible with dbt build (models, tests, seeds all in one command)
Step 2: Set execution mode to WATCHER
Create an ExecutionConfig with execution_mode=ExecutionMode.WATCHER. Optionally set the invocation_mode to InvocationMode.DBT_RUNNER (runs dbt in-process for fastest startup) or leave it to use subprocess mode. The invocation mode controls how the producer task launches the dbt build process.
Key considerations:
- InvocationMode.DBT_RUNNER provides the fastest execution but requires dbt in the same Python environment
- InvocationMode.SUBPROCESS is safer when there might be dependency conflicts
- The Watcher mode automatically creates both producer and consumer tasks
Step 3: Configure render settings
Use RenderConfig to control node selection (select/exclude) and test behavior. Watcher mode respects the same selection syntax as other execution modes. The exclude parameter can filter out specific models from the dbt build.
Key considerations:
- Node selection determines which models appear as consumer tasks in the DAG
- Excluded models are also excluded from the dbt build command
- Test behavior settings apply to how tests are represented in the consumer tasks
Step 4: Set operator arguments
Define operator_args including install_deps (run dbt deps before build), execution_timeout (maximum time for the entire dbt build), and deferrable (whether consumer sensors use async deferral). The deferrable flag is important for resource efficiency: when True, consumer sensors use Airflow's triggerer to watch for completion without occupying a worker slot.
Key considerations:
- deferrable=True (default) makes consumer sensors async, freeing worker slots
- deferrable=False uses synchronous polling, simpler but holds worker slots
- execution_timeout should accommodate the entire dbt build duration
- install_deps runs before the producer starts the dbt build
Step 5: Instantiate the DbtDag or DbtTaskGroup
Create a DbtDag or DbtTaskGroup with the Watcher execution config. On instantiation, Cosmos parses the dbt graph and generates: one DbtProducerWatcherOperator task (the producer) and one DbtConsumerWatcherSensor per dbt node (the consumers). The producer is automatically wired as an upstream dependency of all consumers.
Key considerations:
- Both DbtDag and DbtTaskGroup support Watcher mode
- The producer task appears as a single task in the Airflow UI
- Each consumer task corresponds to one dbt model/test/seed and shows individual status
- The cluster policy plugin can be used to route consumer sensors to the triggerer queue
Step 6: Producer-consumer runtime execution
At runtime, the producer task executes dbt build and streams model completion events to XCom using a background thread (push_xcom_in_background). Each consumer sensor polls XCom (or defers to the triggerer) until its specific model's status appears. When a model completes successfully, the corresponding consumer sensor succeeds. If a model fails, the consumer sensor fails, providing per-model visibility in the Airflow UI.
Key considerations:
- The producer publishes status updates in real-time, not at the end of the build
- Consumer sensors check XCom for their specific model's completion key
- Failed models cause their consumer sensor to fail while other models continue
- The triggerer-based async mode (WatcherTrigger) minimizes resource usage for idle consumers