Environment:Astronomer Astronomer cosmos Cloud Provider Dependencies

Knowledge Sources	astronomer-cosmos Execution Modes
Domains	Infrastructure, Cloud_Computing
Last Updated	2026-02-07 17:00 GMT

Overview

Optional Airflow provider packages for Docker, AWS (ECS/EKS), Azure Container Instance, and GCP Cloud Run Job execution modes.

Description

Cosmos supports running dbt commands in various cloud container environments. Each execution mode requires a specific Airflow provider package that wraps the respective cloud API. These are optional dependencies; only the provider(s) matching the desired execution mode need to be installed. Each provider is guarded by a try/except ImportError block that raises a descriptive error if the package is missing.

Usage

Use this environment when running dbt commands in cloud containers rather than locally. Choose the execution mode matching your infrastructure and install the corresponding provider package.

System Requirements

Category	Requirement	Notes
Docker	apache-airflow-providers-docker >= 3.5.0	For ExecutionMode.DOCKER
AWS EKS	apache-airflow-providers-amazon >= 8.0.0	For ExecutionMode.AWS_EKS
AWS ECS	apache-airflow-providers-amazon >= 8.0.0	For ExecutionMode.AWS_ECS
Azure ACI	apache-airflow-providers-microsoft-azure >= 8.5.0	For ExecutionMode.AZURE_CONTAINER_INSTANCE
GCP Cloud Run	apache-airflow-providers-google >= 10.11.0	For ExecutionMode.GCP_CLOUD_RUN_JOB
Cloud Storage (S3)	apache-airflow-providers-amazon[s3fs] >= 3.0.0	For dbt docs upload to S3
Cloud Storage (GCS)	apache-airflow-providers-google >= 10.17.0, gcsfs < 2025.3.0	For dbt docs upload to GCS
Cloud Storage (Azure)	apache-airflow-providers-microsoft-azure >= 8.5.0	For dbt docs upload to Azure Blob

Dependencies

Python Packages (by execution mode)

Docker: apache-airflow-providers-docker >= 3.5.0
AWS EKS: apache-airflow-providers-amazon >= 8.0.0
AWS ECS: apache-airflow-providers-amazon >= 8.0.0
Azure ACI: apache-airflow-providers-microsoft-azure >= 8.5.0
GCP Cloud Run: apache-airflow-providers-google >= 10.11.0
S3 Storage: apache-airflow-providers-amazon[s3fs] >= 3.0.0
GCS Storage: apache-airflow-providers-google >= 10.17.0, gcsfs < 2025.3.0
Azure Storage: apache-airflow-providers-microsoft-azure >= 8.5.0

Credentials

Docker: Docker daemon access or registry credentials via Airflow connection
AWS (ECS/EKS/S3): aws_default Airflow connection or IAM role
GCP (Cloud Run/GCS): google_cloud_default Airflow connection or service account
Azure (ACI/Blob): wasb_default Airflow connection or managed identity

Quick Install

# Docker execution mode
pip install "astronomer-cosmos[docker]"

# Kubernetes execution mode
pip install "astronomer-cosmos[kubernetes]"

# AWS EKS execution mode
pip install "astronomer-cosmos[aws_eks]"

# AWS ECS execution mode
pip install "astronomer-cosmos[aws-ecs]"

# Azure Container Instance execution mode
pip install "astronomer-cosmos[azure-container-instance]"

# GCP Cloud Run Job execution mode
pip install "astronomer-cosmos[gcp-cloud-run-job]"

# Cloud storage for dbt docs
pip install "astronomer-cosmos[amazon,google,microsoft]"

Code Evidence

Docker provider import guard from cosmos/operators/docker.py:33-39:

try:
    from airflow.providers.docker.operators.docker import DockerOperator
except ImportError:
    raise ImportError(
        "Could not import DockerOperator. Ensure you've installed the docker provider..."
    )

AWS ECS provider import guard from cosmos/operators/aws_ecs.py:38-44:

try:
    from airflow.providers.amazon.aws.operators.ecs import EcsRunTaskOperator
except ImportError:
    raise ImportError(
        "Could not import EcsRunTaskOperator..."
    )

GCS version pin from pyproject.toml:86-87:

# Due to issue https://github.com/fsspec/gcsfs/issues/664
google = ["apache-airflow-providers-google>=10.17.0", "gcsfs<2025.3.0"]

Common Errors

Error Message	Cause	Solution
`ImportError: Could not import DockerOperator`	Docker provider not installed	`pip install "astronomer-cosmos[docker]"`
`ImportError: Could not import EcsRunTaskOperator`	Amazon provider not installed	`pip install "astronomer-cosmos[aws-ecs]"`
`ImportError: Could not import AzureContainerInstancesOperator`	Azure provider not installed	`pip install "astronomer-cosmos[azure-container-instance]"`
`ImportError: Could not import CloudRunExecuteJobOperator`	Google provider not installed	`pip install "astronomer-cosmos[gcp-cloud-run-job]"`
`gcsfs` compatibility error	gcsfs version conflict with fsspec	Pin `gcsfs < 2025.3.0`

Compatibility Notes

Docker: Requires Docker daemon accessible from Airflow worker
AWS ECS/EKS: Same Amazon provider package supports both modes
GCP Cloud Run: Provider >= 10.11.0 required; attribute validation performed at import time
gcsfs: Must be < 2025.3.0 due to fsspec issue (GitHub fsspec/gcsfs#664)

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment