Heuristic:Astronomer Astronomer cosmos Watcher Queue Sizing
| Knowledge Sources | |
|---|---|
| Domains | Optimization, Data_Orchestration |
| Last Updated | 2026-02-07 17:00 GMT |
Overview
Configure a dedicated Airflow queue for Watcher retry execution to handle the resource difference between sensor polling (low CPU/memory) and actual dbt execution (high CPU/memory).
Description
In Watcher execution mode, Cosmos uses a producer-consumer pattern. Consumer tasks start as lightweight sensors polling XCom for status updates (low resource usage). If the producer fails, these consumer tasks retry by executing the individual dbt models directly (high resource usage). This resource profile difference creates a queue sizing challenge: the initial sensor run needs minimal resources, but the retry needs a worker with enough resources to run dbt.
The watcher_dbt_execution_queue setting allows routing producer tasks and consumer retry attempts to a separate Airflow queue backed by workers with larger resource allocations.
Usage
Use this heuristic when running Watcher execution mode with large dbt projects where individual model execution requires more CPU/memory than a sensor poll. Configure a separate queue backed by appropriately-sized workers.
The Insight (Rule of Thumb)
- Action: Set
[cosmos] watcher_dbt_execution_queue = <queue_name>in Airflow config. Ensure workers on that queue have sufficient resources for dbt execution. - Value: The producer task and consumer retries are routed to the specified queue.
- Trade-off: Requires managing two worker pools (sensor pool with small workers and execution pool with large workers).
- Priority: Producer tasks default to priority_weight=20; consumer tasks default to priority_weight=2. This ensures the producer is scheduled before consumers.
Reasoning
From cosmos/settings.py:59-64:
# DBT Watcher Execution Mode Watcher Task Retry Queue
# in watcher mode, if the producer watcher fails, the consumer tasks run the individual
# models on retry. since these tasks are sensors that require low memory/cpu on their
# first try, this setting allows retries to run on a queue with larger resources, which
# is often necessary for larger dbt projects
# this would also be used to run the producer task
watcher_dbt_execution_queue = conf.get(
"cosmos", "watcher_dbt_execution_queue", fallback=None
)
Priority weight constants from cosmos/constants.py:183-184:
CONSUMER_WATCHER_DEFAULT_PRIORITY_WEIGHT = 2
PRODUCER_WATCHER_DEFAULT_PRIORITY_WEIGHT = 20
Trigger rule configuration from cosmos/airflow/graph.py:710-712:
# Make the producer task to be the parent of the root dbt nodes, without blocking
# them from sensing XCom. We only managed to do this in the case of DbtDag.
# The way it is implemented is by setting the trigger_rule to "always" for the
# consumer tasks, and by having the producer task with a high priority_weight.