Principle:Astronomer Astronomer cosmos Airflow Plugin Documentation Hosting

Metadata

Field	Value
Page Type	Principle
Knowledge Sources	Doc (Cosmos Hosting Docs), Repo (astronomer-cosmos)
Domains	Data_Engineering, Documentation, UI
Last Updated	2026-02-07 14:00 GMT

Overview

A principle for serving generated dbt documentation through the Airflow web UI via a custom plugin.

Description

Once dbt docs are uploaded to cloud storage, they need a browsing interface. Rather than requiring a separate web server, the dbt docs can be served directly through Airflow's web UI via a plugin. The plugin reads docs from cloud storage and renders them in an iframe. This provides a single-pane-of-glass experience where both pipeline monitoring and documentation are accessible from the same interface.

The Plugin Approach

Airflow's plugin system allows extending the web UI with custom views, menu items, and API endpoints. The Cosmos plugin leverages this mechanism to:

1. Register a new menu item under the Browse category in the Airflow navigation.
2. Create a web endpoint that fetches documentation artifacts from the configured cloud storage path.
3. Render the dbt documentation within an iframe embedded in the Airflow UI.

This approach eliminates the need for a dedicated documentation hosting server, reducing infrastructure complexity and operational overhead.

Version-Specific Implementations

Airflow's web framework has evolved across major versions, requiring version-specific plugin implementations:

• Airflow 2.x: The web UI is built on Flask and Flask AppBuilder (FAB). The Cosmos plugin registers a FAB BaseView subclass that handles HTTP requests and renders templates with Jinja2.
• Airflow 3.x: The web UI transitions to FastAPI. The Cosmos plugin creates a FastAPI application with dedicated route handlers, mounted at a URL prefix within the Airflow server.

Both implementations serve the same purpose but use different web frameworks and routing conventions. The plugin system automatically selects the appropriate implementation based on the detected Airflow version.

Storage Integration

The plugin reads documentation from cloud storage using the same Airflow connections that were used for uploading. It supports:

• S3: Using S3Hook to read objects from S3 buckets.
• GCS: Using GCSHook to read objects from GCS buckets.
• Azure Blob Storage: Using WasbHook to read blobs from Azure containers.

The storage type is automatically detected from the configured documentation path (e.g., s3://bucket/path, gs://bucket/path, wasb://container/path).

Usage

Use Airflow plugin documentation hosting when:

• Single-pane-of-glass access: Team members should be able to access dbt documentation directly from the Airflow UI without navigating to a separate URL or deploying a separate docs hosting solution.
• Existing Airflow infrastructure: The organization already uses Airflow as its primary orchestration interface, and adding documentation hosting to the same interface reduces context switching.
• Minimal infrastructure overhead: Deploying and maintaining a separate web server for documentation hosting is undesirable. The plugin approach reuses the existing Airflow web server.
• Access control integration: Airflow's built-in authentication and authorization controls apply to the documentation endpoint, ensuring consistent access management.

Theoretical Basis

Airflow's plugin system allows extending the UI with custom views. The Cosmos plugin registers a new menu item that loads dbt documentation from a configured cloud storage path.

Airflow Plugin Architecture

Airflow discovers plugins through its plugin manager, which scans the airflow_plugins entry point and registered Python packages. Each plugin class inherits from AirflowPlugin and declares:

• name: A unique identifier for the plugin.
• appbuilder_views (Airflow 2.x): A list of Flask AppBuilder view registrations, each specifying a view class, a display name, and a navigation category.
• fastapi_apps (Airflow 3.x): A list of FastAPI application registrations, each specifying an app instance and a URL prefix.

Iframe-Based Rendering

The plugin serves dbt documentation within an iframe rather than directly integrating the HTML into the Airflow UI. This approach:

• Isolates CSS and JavaScript: The dbt docs application has its own styles and scripts that could conflict with Airflow's UI framework. An iframe provides complete isolation.
• Preserves the full docs experience: The dbt documentation UI includes search, lineage visualization, and interactive navigation that work best in their own rendering context.
• Simplifies implementation: The plugin only needs to fetch and serve the raw HTML file; no transformation or adaptation is required.

Configuration-Driven Behavior

The plugin reads its configuration from the [cosmos] section of airflow.cfg, including:

• dbt_docs_dir: The cloud storage path where documentation artifacts are stored.
• dbt_docs_conn_id: The Airflow connection ID for authenticating with the storage provider.
• dbt_docs_index_file_name: The name of the index file to serve (defaults to index.html, can be static_index.html for static docs).

Related Pages

• Implementation:Astronomer_Astronomer_cosmos_Cosmos_Plugin