Environment:TobikoData Sqlmesh Dbt Compatibility

Knowledge Sources	TobikoData/sqlmesh
Domains	dbt, Data Transformation, Migration
Last Updated	2026-02-07 21:00 GMT

Overview

dbt project compatibility environment enabling SQLMesh to run existing dbt projects while providing advanced features like virtual environments.

Description

The dbt compatibility environment allows SQLMesh to natively execute dbt projects without conversion, providing full support for dbt models, seeds, sources, tests, snapshots, and macros. The integration includes the DbtLoader for project discovery and manifest parsing, dbt adapter system integration, and Jinja template processing with macro support. Multiple engine-specific dbt adapters are available including BigQuery, DuckDB, Snowflake, Athena, ClickHouse, Databricks, Redshift, and Trino.

Usage

This environment is required when running existing dbt projects through SQLMesh, initializing SQLMesh in dbt projects with 'sqlmesh init -t dbt', converting dbt projects to native SQLMesh format, or leveraging SQLMesh's virtual environments and plan/apply workflows with dbt models. Requires dbt profiles.yml for database connections.

System Requirements

Category	Requirement	Notes
Python	>= 3.9, < 3.12 recommended	For dbt-core compatibility
dbt Profiles	profiles.yml configured	Standard dbt connection setup
Database	Supported dbt adapter	One of 8+ supported engines
importlib-metadata	Required for Python < 3.12	Package discovery

Dependencies

System Packages

• Database-specific client libraries (per dbt adapter requirements)
• Git (for dbt package management)

Python Packages

• dbt-core<2 - Core dbt functionality (version 1.x)

Engine-specific dbt adapters (optional, install per engine):

• dbt-bigquery - Google BigQuery adapter
• dbt-duckdb>=1.7.1 - DuckDB adapter
• dbt-snowflake!=1.10.1 - Snowflake adapter (exclude buggy 1.10.1)
• dbt-athena-community - AWS Athena adapter
• dbt-clickhouse - ClickHouse adapter
• dbt-databricks - Databricks adapter
• dbt-redshift - Amazon Redshift adapter
• dbt-trino - Trino adapter

Note: dbt-snowflake 1.10.1 has a dependency bug requiring dbt-adapters>=1.16.6 for InvalidCatalogIntegrationConfigError

Credentials

dbt environment variables:

• DBT_PROFILES_DIR - Custom location for profiles.yml (default: ~/.dbt)
• DBT_MACRO_DEBUGGING - Enable macro debugging output

Database credentials configured via profiles.yml following standard dbt patterns:

• Snowflake: account, user, password, role, warehouse, database
• BigQuery: project, dataset, keyfile or method
• Databricks: host, http_path, token, catalog
• Others: per dbt adapter documentation

SQLMesh-specific variables:

• SQLMESH_HOME - SQLMesh configuration directory
• SQLMESH_GATEWAY - Gateway selection for dbt project

Quick Install

# Install SQLMesh with dbt support
pip install "sqlmesh[dbt]"

# Install specific dbt adapter
pip install dbt-snowflake  # or dbt-bigquery, dbt-duckdb, etc.

# Initialize SQLMesh in existing dbt project
cd my-dbt-project
sqlmesh init -t dbt

# Configure profiles.yml (standard dbt setup)
cat > ~/.dbt/profiles.yml << EOF
my_project:
  target: dev
  outputs:
    dev:
      type: snowflake
      account: myaccount
      user: myuser
      password: mypassword
      warehouse: compute_wh
      database: analytics
      schema: dbt_dev
EOF

# Run dbt project through SQLMesh
sqlmesh plan
sqlmesh apply

# Convert dbt project to SQLMesh format (optional)
sqlmesh dbt convert

Code Evidence

# File: pyproject.toml:108
dbt = ["dbt-core<2"]

# File: pyproject.toml:64-67
# Known issue with dbt-snowflake 1.10.1
# dbt-snowflake==1.10.1 depends on InvalidCatalogIntegrationConfigError
# which only exists in dbt-adapters>=1.16.6, but the package metadata
# declares dbt-adapters<1.16.0 as valid, causing pip to fail
"dbt-snowflake!=1.10.1",

# File: sqlmesh/dbt/profile.py:72
def load_profile(profile_name: str) -> t.Dict[str, t.Any]:
    """Load dbt profile from profiles.yml."""
    profiles_dir = os.environ.get("DBT_PROFILES_DIR", os.path.expanduser("~/.dbt"))
    profile_path = os.path.join(profiles_dir, "profiles.yml")

    with open(profile_path) as f:
        profiles = yaml.safe_load(f)

    return profiles.get(profile_name, {})

# File: sqlmesh/dbt/builtin.py:434
def enable_macro_debugging() -> bool:
    """Check if macro debugging is enabled."""
    return os.environ.get("DBT_MACRO_DEBUGGING", "").lower() in ("1", "true", "t")

# File: sqlmesh/dbt/loader.py:38
# Uses importlib_metadata for Python < 3.12
try:
    from importlib.metadata import entry_points
except ImportError:
    from importlib_metadata import entry_points  # type: ignore

class DbtLoader:
    """Loads dbt projects into SQLMesh context."""

    def load(self, project_dir: str) -> Context:
        """Load dbt project and return SQLMesh context."""
        # ... implementation

Common Errors

Error Message	Cause	Solution
ModuleNotFoundError: No module named 'dbt'	dbt-core not installed	Install with pip install "sqlmesh[dbt]"
Could not find profile	profiles.yml missing or misconfigured	Create ~/.dbt/profiles.yml with valid profile
InvalidCatalogIntegrationConfigError	Using dbt-snowflake 1.10.1	Upgrade/downgrade: pip install "dbt-snowflake!=1.10.1"
Adapter not found	Engine-specific dbt adapter not installed	Install adapter: pip install dbt-{engine}
Macro compilation failed	Unsupported Jinja syntax	Check macro for SQLMesh compatibility
importlib_metadata not found	Python < 3.12 missing dependency	Install pip install importlib-metadata

Compatibility Notes

• dbt-core version 1.x supported (v2 not yet supported)
• dbt-snowflake 1.10.1 must be excluded due to dependency resolution bug
• dbt-duckdb requires >= 1.7.1 for latest features
• All standard dbt materializations supported (table, view, incremental, ephemeral)
• Incremental strategies supported: delete+insert, merge, insert_overwrite
• SCD Type 2 snapshots fully supported with timestamp and check strategies
• Jinja templating and macros converted to SQLMesh equivalents
• dbt tests integrated into SQLMesh test framework
• Package dependencies resolved using dbt's packages.yml
• profiles.yml used for connection configuration (standard dbt)
• DBT_PROFILES_DIR environment variable respected
• Macro debugging available via DBT_MACRO_DEBUGGING flag
• importlib_metadata required for Python < 3.12 package discovery
• Full support for dbt sources, seeds, and snapshots
• Cross-database compatibility via SQLMesh's multi-dialect support

Related Pages

• Implementation:TobikoData_Sqlmesh_ManifestHelper_Init
• Implementation:TobikoData_Sqlmesh_DbtLoader_Load
• Implementation:TobikoData_Sqlmesh_DbtContext_Add_Macros
• Implementation:TobikoData_Sqlmesh_ModelConfig_To_Sqlmesh
• Implementation:TobikoData_Sqlmesh_Context_Plan_Test_For_Dbt
• Implementation:TobikoData_Sqlmesh_Init_Example_Project