Implementation:Astronomer Astronomer cosmos Cloud Storage Connection Setup
Metadata
| Field | Value |
|---|---|
| Page Type | Implementation (External Tool Doc) |
| Knowledge Sources | Doc (Airflow Connections), Repo (astronomer-cosmos) |
| Domains | Configuration, Cloud_Storage, Security |
| Last Updated | 2026-02-07 14:00 GMT |
Overview
Configuration guide for setting up Airflow connections to cloud storage services used by astronomer-cosmos documentation operators. This is an external tool documentation page covering the Airflow connection infrastructure that the Cosmos documentation operators depend on.
Description
The astronomer-cosmos documentation operators (DbtDocsS3LocalOperator, DbtDocsGCSLocalOperator, DbtDocsAzureStorageLocalOperator) and the Cosmos plugin all require Airflow connections to authenticate with cloud storage providers. This page documents how to configure these connections.
Source Location
External (Airflow core): airflow.models.connection.Connection
External Dependencies
| Provider | Package | Purpose |
|---|---|---|
| Amazon S3 | apache-airflow-providers-amazon |
Provides S3Hook for interacting with S3 buckets
|
| Google Cloud Storage | apache-airflow-providers-google |
Provides GCSHook for interacting with GCS buckets
|
| Microsoft Azure Blob Storage | apache-airflow-providers-microsoft-azure |
Provides WasbHook for interacting with Azure Blob containers
|
Interface
Connections are created through the Airflow Admin UI or the Airflow CLI.
UI Method
Navigate to Admin > Connections > Add Connection (or the "+" button) in the Airflow web interface.
S3 Connection
| Field | Value |
|---|---|
| Connection Id | aws_default (or a custom ID)
|
| Connection Type | Amazon Web Services |
| AWS Access Key ID | Your AWS access key |
| AWS Secret Access Key | Your AWS secret key |
| Extra | {"region_name": "us-east-1"} (optional, for region-specific endpoints)
|
GCS Connection
| Field | Value |
|---|---|
| Connection Id | google_cloud_default (or a custom ID)
|
| Connection Type | Google Cloud |
| Keyfile JSON | Paste the full contents of your service account JSON key file |
| Project Id | Your GCP project ID |
| Scopes | https://www.googleapis.com/auth/cloud-platform (optional)
|
Azure Blob Storage Connection
| Field | Value |
|---|---|
| Connection Id | wasb_default (or a custom ID)
|
| Connection Type | Azure Blob Storage |
| Login | Your Azure storage account name |
| Password | Your Azure storage account key or SAS token |
I/O Contract
Inputs
| Input | Type | Description |
|---|---|---|
| Cloud provider credentials | Various | AWS access keys, GCP service account JSON, or Azure storage account key/SAS token. The specific format depends on the provider. |
Outputs
| Output | Type | Description |
|---|---|---|
| Airflow connection ID | str | A referenceable connection ID (e.g., aws_default, google_cloud_default, wasb_default) usable by DbtDocsCloudLocalOperator subclasses and the Cosmos plugin.
|
Usage Examples
CLI: Create an S3 Connection
airflow connections add aws_default \
--conn-type aws \
--conn-login "AKIAIOSFODNN7EXAMPLE" \
--conn-password "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \
--conn-extra '{"region_name": "us-east-1"}'
CLI: Create a GCS Connection
airflow connections add google_cloud_default \
--conn-type google_cloud_platform \
--conn-extra '{
"key_path": "/path/to/service-account-key.json",
"project": "my-gcp-project"
}'
Alternatively, using an inline keyfile:
airflow connections add google_cloud_default \
--conn-type google_cloud_platform \
--conn-extra '{
"keyfile_dict": {
"type": "service_account",
"project_id": "my-gcp-project",
"private_key_id": "key-id",
"private_key": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----\n",
"client_email": "my-sa@my-gcp-project.iam.gserviceaccount.com",
"client_id": "123456789",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token"
}
}'
CLI: Create an Azure Blob Storage Connection
airflow connections add wasb_default \
--conn-type wasb \
--conn-login "mystorageaccount" \
--conn-password "your-storage-account-key-here"
Environment Variable Method
Connections can also be defined as environment variables. The format is AIRFLOW_CONN_{CONN_ID} with a URI-encoded connection string:
# S3 connection via environment variable
export AIRFLOW_CONN_AWS_DEFAULT="aws://AKIAIOSFODNN7EXAMPLE:wJalrXUtnFEMI%2FK7MDENG%2FbPxRfiCYEXAMPLEKEY@/?region_name=us-east-1"
# Azure Blob Storage via environment variable
export AIRFLOW_CONN_WASB_DEFAULT="wasb://mystorageaccount:your-storage-account-key@"
Using the Connection with Cosmos Operators
Once the connection is configured, reference it by ID in the Cosmos documentation operators:
from cosmos.operators.local import DbtDocsS3LocalOperator
upload_docs = DbtDocsS3LocalOperator(
task_id="upload_docs",
project_dir="/usr/local/airflow/dags/dbt/my_project",
profile_config=profile_config,
connection_id="aws_default", # References the connection created above
bucket_name="my-dbt-docs-bucket",
)
Verifying a Connection
Test that a connection is properly configured using the Airflow CLI:
# List all connections
airflow connections list
# Test a specific connection (Airflow 2.7+)
airflow connections test aws_default