Environment:Astronomer Astronomer cosmos Kubernetes Provider

Knowledge Sources	astronomer-cosmos Kubernetes Execution Mode
Domains	Infrastructure, Container_Orchestration
Last Updated	2026-02-07 17:00 GMT

Overview

Apache Airflow Kubernetes provider >= 7.14.0 environment with version-specific log handling for running dbt commands inside Kubernetes pods.

Description

This environment provides the Kubernetes execution mode for Cosmos. It requires the apache-airflow-providers-cncf-kubernetes package at version 7.14.0 or higher. The Watcher execution mode for Kubernetes has a stricter requirement of provider version >= 10.8.0, with enhanced log streaming features available at >= 10.10.0. Cosmos includes a custom PodLogManager override to work around a known Airflow bug with repeating Kubernetes pod logs (apache/airflow#59366).

Usage

Use this environment when running dbt commands in Kubernetes pods via ExecutionMode.KUBERNETES or ExecutionMode.WATCHER_KUBERNETES. It is the mandatory prerequisite for the DbtKubernetesBaseOperator and related operators.

System Requirements

Category	Requirement	Notes
Kubernetes Cluster	Any supported K8s version	Must be reachable from Airflow workers
Airflow Provider	apache-airflow-providers-cncf-kubernetes >= 7.14.0	For standard K8s execution
Airflow Provider (Watcher)	apache-airflow-providers-cncf-kubernetes >= 10.8.0	For Watcher-Kubernetes mode
Container Image	Docker image with dbt installed	Must include the required dbt adapter

Dependencies

Python Packages

• apache-airflow-providers-cncf-kubernetes >= 7.14.0 (standard)
• apache-airflow-providers-cncf-kubernetes >= 10.8.0 (Watcher mode minimum)
• apache-airflow-providers-cncf-kubernetes >= 10.10.0 (enhanced log streaming)

Credentials

• Kubernetes connection: An Airflow connection of type kubernetes or a valid kubeconfig
• Container registry credentials: If using a private Docker registry for the dbt image

Quick Install

pip install "astronomer-cosmos[kubernetes]"

Code Evidence

Minimum Watcher K8s provider version constant from cosmos/constants.py:200:

_K8s_WATCHER_MIN_K8S_PROVIDER_VERSION = Version("10.8.0")

Provider version validation from cosmos/airflow/_override.py:61-71:

if k8s_provider_version >= Version("10.10.0"):
    parsed = PodLogHandler.parse_log_line(line)
elif k8s_provider_version >= _K8s_WATCHER_MIN_K8S_PROVIDER_VERSION:
    try:
        parsed = super()._parse_log_line(line)
    except AttributeError:
        parsed = parse_log_line(line)
else:
    raise ValueError(
        f"Kubernetes provider version {k8s_provider_version} is not supported for Watcher execution mode. "
        f"Minimum required version is {_K8s_WATCHER_MIN_K8S_PROVIDER_VERSION}"
    )

K8s pod log workaround from cosmos/airflow/_override.py:19-21:

# This is being added to overcome the issue with the KubernetesPodOperator logs repeating:
# https://github.com/apache/airflow/issues/59366
# It can be removed once it is fixed in the upstream provider.

Connection timeout configuration from cosmos/airflow/_override.py:76-85:

# We timeout connections after 30 minutes because otherwise they can get
# stuck forever. The 30 is somewhat arbitrary.
connection_timeout = 60 * 30
# We set a shorter read timeout because that helps reduce *connection* timeouts
read_timeout = 60 * 5

Common Errors

Error Message	Cause	Solution
ImportError: Could not import KubernetesPodOperator	Kubernetes provider not installed	pip install apache-airflow-providers-cncf-kubernetes>=7.14.0
ValueError: Kubernetes provider version X is not supported for Watcher execution mode	Provider version below 10.8.0 for Watcher mode	Upgrade to >= 10.8.0
Repeating pod logs	Known Airflow bug (apache/airflow#59366)	Cosmos includes a built-in workaround via PodLogManager
Connection timeout after 30 minutes	K8s API connection stuck	Expected behavior; connection will be retried automatically

Compatibility Notes

• Provider >= 10.10.0: Uses PodLogHandler.parse_log_line for log parsing (newer API)
• Provider >= 10.8.0 and < 10.10.0: Falls back to _parse_log_line or parse_log_line
• Provider < 10.8.0: Not supported for Watcher execution mode; raises ValueError
• Airflow 3.0+: Kubernetes provider imports may shift to the SDK module namespace

Related Pages

• Implementation:Astronomer_Astronomer_cosmos_DbtKubernetesOperator_Execute