Principle:Dagster io Dagster BI Tool Integration
| Field | Value |
|---|---|
| Principle Name | BI Tool Integration |
| Category | Data Lineage |
| Domains | Business_Intelligence, Analytics |
| Repository | dagster-io/dagster |
Overview
Strategy for integrating business intelligence tools into Dagster's asset graph to provide end-to-end lineage from data sources through transformations to dashboards.
Description
BI tool integration connects visualization platforms (Power BI, Tableau, Looker) to Dagster by representing reports, semantic models, and dashboards as Dagster assets. Custom translators map BI tool metadata (table references, data sources) to upstream Dagster asset dependencies, creating a complete lineage graph from raw data ingestion through transformation to final visualization. This enables data teams to see the full impact of upstream changes on downstream dashboards.
The integration works through several key mechanisms:
- Asset discovery -- The integration queries the BI tool's API to enumerate all reports, semantic models, dashboards, and data sources in a workspace.
- Dependency mapping -- Custom translators (
DagsterPowerBITranslator,DagsterTableauTranslator) convert BI tool metadata into DagsterAssetSpecobjects with proper upstream dependencies. - Lineage bridging -- By mapping BI table references to Dagster asset keys, the integration creates a unified lineage graph spanning from data ingestion through transformation to visualization.
- Freshness tracking -- BI assets participate in Dagster's freshness policies, enabling monitoring of when dashboard data was last refreshed.
Usage
Use when your data pipeline feeds BI dashboards and you need end-to-end lineage visibility. The integration surfaces BI artifacts as Dagster assets with proper dependency links to upstream data models. This is valuable for:
- Impact analysis -- Determine which dashboards are affected when an upstream table schema changes or data quality degrades.
- Freshness monitoring -- Track when BI dashboard data was last refreshed and set policies for acceptable staleness.
- Unified catalog -- Provide a single view of all data assets from raw sources through transformations to dashboards.
- Governance -- Ensure all BI reports are traceable to their source data for compliance and audit purposes.
Theoretical Basis
BI integration implements the bridge pattern between data engineering and analytics layers. By modeling BI artifacts as assets in the same graph as data assets, the system creates a unified dependency graph. Custom translators map between BI tool naming conventions and Dagster asset keys. This enables impact analysis (what dashboards break if this table changes?) and freshness tracking (when was this dashboard's data last updated?).
The design follows these principles:
- Bridge pattern -- Translators bridge the conceptual gap between BI tool representations (reports, semantic models) and Dagster representations (assets with dependencies).
- Open-closed principle -- The base translator provides default behavior; users extend it by subclassing to customize asset key mapping, group names, or dependency resolution without modifying the library.
- Single source of truth -- The BI tool's own API is the source of truth for what reports and models exist, ensuring the Dagster graph stays synchronized.