Principle:Dagster io Dagster Dbt Asset Translation
| Attribute | Value |
|---|---|
| Title | Dbt Asset Translation |
| Category | Principle |
| Domains | Data_Engineering, dbt |
| Repository | Dagster_io_Dagster |
Overview
Mechanism for mapping dbt model metadata (sources, refs, tests) to Dagster asset specifications with customizable key translation.
Description
dbt asset translation converts dbt's manifest metadata into Dagster asset and check specs. The translator maps dbt source() references to upstream Dagster asset keys, dbt ref() references to inter-model dependencies, and dbt test declarations to Dagster asset checks. Custom translator subclasses can override default behavior to map dbt models to specific asset key schemes, groups, and dependency patterns.
The translation process operates at multiple levels:
- Asset key translation: The
get_asset_key()method converts a dbt resource's fully qualified name into a DagsterAssetKey. By default, model keys use the model name; sources usesource_name/table_name. - Dependency resolution: The
get_asset_spec()method recursively resolves upstream unique IDs from the dbt manifest, translating each into anAssetDepwith the correct Dagster asset key. - Group assignment: The
get_group_name()method derives Dagster group names from dbt resource properties (e.g., dbt groups, folder structure, or custom logic). - Check translation: The
get_asset_check_spec()method converts dbt tests intoAssetCheckSpecobjects linked to the appropriate asset.
Usage
Use when the default dbt-to-Dagster mapping does not match your asset key scheme, when you need custom group assignments, or when dbt sources should map to specific upstream Dagster assets (e.g., ingestion assets).
Common customization scenarios:
- Adding a prefix or namespace to all dbt-generated asset keys
- Remapping dbt source references to match existing Dagster ingestion assets
- Assigning dbt models to Dagster groups based on dbt tags or folder structure
- Controlling which dbt tests become Dagster asset checks via
DagsterDbtTranslatorSettings
Theoretical Basis
The translator implements the mediator pattern between two metadata schemas. dbt's manifest JSON describes models, sources, and tests in dbt's namespace. Dagster requires AssetSpec objects in its namespace. The translator converts between these schemas, with customization points at each conversion step.
Key design decisions:
- Memoized resolution: The
get_asset_spec()method caches resolved specs by(manifest_id, unique_id, project_id)to avoid redundant computation during recursive dependency resolution. - Extensibility via inheritance: Individual translation methods (
get_asset_key,get_group_name,get_description, etc.) can be overridden independently, allowing selective customization. - Settings-based feature flags:
DagsterDbtTranslatorSettingsprovides boolean flags (e.g.,enable_asset_checks,enable_code_references) to opt into or out of specific translation behaviors without subclassing. - Jinja-based key templates: For the component system, key templates provide string-level customization without Python code.