Implementation:Astronomer Astronomer cosmos ExecutionConfig Init

Metadata

Field	Value
Page Type	Implementation
Repository	astronomer-cosmos
Source File	`cosmos/config.py`, Lines L402-451
Implements	Principle:Astronomer_Astronomer_cosmos_Execution_Configuration
Import	`from cosmos.config import ExecutionConfig`

Overview

ExecutionConfig is the concrete tool provided by the astronomer-cosmos library for configuring dbt runtime execution mode and settings. It is a dataclass that encapsulates all preferences governing how dbt commands are executed at runtime, including the execution mode (local, container, Kubernetes), the invocation mode (dbt runner, subprocess), the dbt executable path, and runtime-specific filesystem paths.

This configuration is consumed by cosmos's task execution engine to determine where and how each dbt command is run.

Source Location

Property	Value
Repository	astronomer-cosmos
File	`cosmos/config.py`
Lines	L402-451
Class	`ExecutionConfig` (decorated with `@dataclass`)

Code Reference

Signature

@dataclass
class ExecutionConfig:
    execution_mode: ExecutionMode = ExecutionMode.LOCAL
    invocation_mode: InvocationMode | None = None
    test_indirect_selection: TestIndirectSelection = TestIndirectSelection.EAGER
    dbt_executable_path: str | Path = field(default_factory=get_system_dbt)
    dbt_project_path: InitVar[str | Path | None] = None
    virtualenv_dir: str | Path | None = None
    project_path: Path | None = field(init=False)
    async_py_requirements: list[str] | None = None
    setup_operator_args: dict[str, Any] | None = None

Import Statements

from cosmos.config import ExecutionConfig
from cosmos.constants import ExecutionMode, InvocationMode, TestIndirectSelection

Parameters

Parameter	Type	Default	Description
`execution_mode`	`ExecutionMode`	`ExecutionMode.LOCAL`	The runtime environment where dbt commands execute.
`invocation_mode`	None	`None`	How dbt is called within the execution environment (`DBT_RUNNER` or `SUBPROCESS`). `None` uses the default for the execution mode.
`test_indirect_selection`	`TestIndirectSelection`	`EAGER`	Strategy for selecting tests when specific models are selected.
`dbt_executable_path`	Path	`get_system_dbt()`	Path to the dbt executable in the runtime environment.
`dbt_project_path`	Path \| None]	`None`	Init-only variable specifying the dbt project path in the runtime environment.
`virtualenv_dir`	Path \| None	`None`	Directory for creating virtual environments (used with `VIRTUALENV` execution mode).
`project_path`	None	(computed)	Resolved project path, computed from `dbt_project_path` during `__post_init__`.
`async_py_requirements`	None	`None`	Python package requirements for async execution environments.
`setup_operator_args`	None	`None`	Additional arguments passed to the setup operator for containerized execution modes.

ExecutionMode Enum Values

Value	Description
`LOCAL`	Execute dbt directly on the Airflow worker.
`VIRTUALENV`	Execute dbt in an isolated Python virtual environment on the worker.
`DOCKER`	Execute dbt inside a Docker container.
`KUBERNETES`	Execute dbt as a Kubernetes pod.
`AWS_EKS`	Execute dbt on Amazon Elastic Kubernetes Service.
`AWS_ECS`	Execute dbt on Amazon Elastic Container Service.
`AZURE_CONTAINER_INSTANCE`	Execute dbt on Azure Container Instances.
`GCP_CLOUD_RUN_JOB`	Execute dbt on Google Cloud Run Jobs.
`AIRFLOW_ASYNC`	Execute dbt asynchronously via Airflow's async mechanism.
`WATCHER`	Execute dbt with a watcher pattern for long-running tasks.
`WATCHER_KUBERNETES`	Execute dbt with the watcher pattern on Kubernetes.

I/O Contract

Inputs

All parameters are optional with sensible defaults for local execution. The primary inputs are:

Execution Mode: Determines the runtime environment. This is the most impactful setting as it changes the entire execution architecture.
Invocation Mode: Fine-tunes how dbt is called within the chosen execution mode.
Runtime Paths: dbt_executable_path and dbt_project_path specify filesystem paths as they exist in the runtime environment (which may differ from the rendering environment).

Outputs

An ExecutionConfig instance that provides:

All execution preferences as typed attributes.
The resolved project_path attribute (computed from the init-only dbt_project_path).
Configuration consumed by cosmos's operator factory to generate the appropriate Airflow operator for each dbt task.

Post-Initialization

During __post_init__, the following processing occurs:

dbt_project_path (init-only) is resolved into project_path as a Path object.
Validation ensures the configuration is internally consistent (e.g., virtualenv_dir is only meaningful with VIRTUALENV execution mode).

Usage Examples

Local Execution (Default)

from cosmos.config import ExecutionConfig
from cosmos.constants import ExecutionMode

# Simple local execution (default settings)
execution_config = ExecutionConfig(
    execution_mode=ExecutionMode.LOCAL,
)

This is the simplest configuration, suitable for development environments or setups where dbt is installed directly on the Airflow worker.

Kubernetes Execution

from cosmos.config import ExecutionConfig
from cosmos.constants import ExecutionMode

# Execute dbt as Kubernetes pods
execution_config = ExecutionConfig(
    execution_mode=ExecutionMode.KUBERNETES,
    dbt_project_path="/dbt/my_project",
    dbt_executable_path="/usr/local/bin/dbt",
)

In Kubernetes mode, dbt_project_path and dbt_executable_path refer to paths inside the container image, not on the Airflow scheduler's filesystem. The container image must include dbt, the project files, and all necessary adapters.

Watcher Mode Execution

from cosmos.config import ExecutionConfig
from cosmos.constants import ExecutionMode

# Execute dbt with watcher pattern for long-running tasks
execution_config = ExecutionConfig(
    execution_mode=ExecutionMode.WATCHER,
    dbt_project_path="/dbt/my_project",
    dbt_executable_path="/usr/local/bin/dbt",
)

The watcher mode is designed for long-running dbt commands where the Airflow task should not block a worker slot. Instead, a lightweight watcher process monitors the dbt execution and reports completion.

Virtual Environment Execution

from cosmos.config import ExecutionConfig
from cosmos.constants import ExecutionMode

# Execute dbt in an isolated virtual environment
execution_config = ExecutionConfig(
    execution_mode=ExecutionMode.VIRTUALENV,
    virtualenv_dir="/tmp/cosmos_venvs",
    async_py_requirements=["dbt-core==1.7.0", "dbt-postgres==1.7.0"],
)

Virtual environment execution is useful when multiple dbt projects require different dbt versions on the same Airflow cluster.

Complete DAG Integration

from cosmos import DbtTaskGroup, ProjectConfig, ProfileConfig, RenderConfig, ExecutionConfig
from cosmos.constants import ExecutionMode, LoadMode
from cosmos.profiles.postgres.user_pass import PostgresUserPasswordProfileMapping

# Render from manifest (fast), execute on Kubernetes (isolated)
project_config = ProjectConfig(
    dbt_project_path="/usr/local/airflow/dags/dbt/my_project",
    manifest_path="/usr/local/airflow/dags/dbt/target/manifest.json",
)

profile_config = ProfileConfig(
    profile_name="my_project",
    target_name="prod",
    profile_mapping=PostgresUserPasswordProfileMapping(
        conn_id="postgres_prod",
    ),
)

render_config = RenderConfig(
    load_method=LoadMode.MANIFEST,
    select=["path:models/marts"],
)

execution_config = ExecutionConfig(
    execution_mode=ExecutionMode.KUBERNETES,
    dbt_project_path="/dbt/my_project",
    dbt_executable_path="/usr/local/bin/dbt",
)

dbt_tasks = DbtTaskGroup(
    project_config=project_config,
    profile_config=profile_config,
    render_config=render_config,
    execution_config=execution_config,
)

This example demonstrates the dual-path architecture: the project is parsed from a local manifest during rendering, but dbt commands execute inside Kubernetes pods at runtime with different filesystem paths.

Related Pages

Implements Principle

Principle:Astronomer_Astronomer_cosmos_Execution_Configuration — The principle this implementation realizes.

Complementary Implementations

Implementation:Astronomer_Astronomer_cosmos_ProjectConfig_Init — Complementary implementation for project path configuration.
Implementation:Astronomer_Astronomer_cosmos_ProfileConfig_Init — Complementary implementation for profile configuration.
Implementation:Astronomer_Astronomer_cosmos_RenderConfig_Init — Complementary implementation for graph rendering configuration.

Requires Environment

Environment:Astronomer_Astronomer_cosmos_Python_Airflow_Runtime

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment