Environment:TobikoData Sqlmesh Dbt Compatibility

Knowledge Sources	TobikoData/sqlmesh
Domains	dbt, Data Transformation, Migration
Last Updated	2026-02-07 21:00 GMT

Overview

dbt project compatibility environment enabling SQLMesh to run existing dbt projects while providing advanced features like virtual environments.

Description

The dbt compatibility environment allows SQLMesh to natively execute dbt projects without conversion, providing full support for dbt models, seeds, sources, tests, snapshots, and macros. The integration includes the DbtLoader for project discovery and manifest parsing, dbt adapter system integration, and Jinja template processing with macro support. Multiple engine-specific dbt adapters are available including BigQuery, DuckDB, Snowflake, Athena, ClickHouse, Databricks, Redshift, and Trino.

Usage

This environment is required when running existing dbt projects through SQLMesh, initializing SQLMesh in dbt projects with 'sqlmesh init -t dbt', converting dbt projects to native SQLMesh format, or leveraging SQLMesh's virtual environments and plan/apply workflows with dbt models. Requires dbt profiles.yml for database connections.

System Requirements

Category	Requirement	Notes
Python	>= 3.9, < 3.12 recommended	For dbt-core compatibility
dbt Profiles	profiles.yml configured	Standard dbt connection setup
Database	Supported dbt adapter	One of 8+ supported engines
importlib-metadata	Required for Python < 3.12	Package discovery

Dependencies

System Packages

Database-specific client libraries (per dbt adapter requirements)
Git (for dbt package management)

Python Packages

dbt-core<2 - Core dbt functionality (version 1.x)

Engine-specific dbt adapters (optional, install per engine):

dbt-bigquery - Google BigQuery adapter
dbt-duckdb>=1.7.1 - DuckDB adapter
dbt-snowflake!=1.10.1 - Snowflake adapter (exclude buggy 1.10.1)
dbt-athena-community - AWS Athena adapter
dbt-clickhouse - ClickHouse adapter
dbt-databricks - Databricks adapter
dbt-redshift - Amazon Redshift adapter
dbt-trino - Trino adapter

Note: dbt-snowflake 1.10.1 has a dependency bug requiring dbt-adapters>=1.16.6 for InvalidCatalogIntegrationConfigError

Credentials

dbt environment variables:

DBT_PROFILES_DIR - Custom location for profiles.yml (default: ~/.dbt)
DBT_MACRO_DEBUGGING - Enable macro debugging output

Database credentials configured via profiles.yml following standard dbt patterns:

Snowflake: account, user, password, role, warehouse, database
BigQuery: project, dataset, keyfile or method
Databricks: host, http_path, token, catalog
Others: per dbt adapter documentation

SQLMesh-specific variables:

SQLMESH_HOME - SQLMesh configuration directory
SQLMESH_GATEWAY - Gateway selection for dbt project

Quick Install

# Install SQLMesh with dbt support
pip install "sqlmesh[dbt]"

# Install specific dbt adapter
pip install dbt-snowflake  # or dbt-bigquery, dbt-duckdb, etc.

# Initialize SQLMesh in existing dbt project
cd my-dbt-project
sqlmesh init -t dbt

# Configure profiles.yml (standard dbt setup)
cat > ~/.dbt/profiles.yml << EOF
my_project:
  target: dev
  outputs:
    dev:
      type: snowflake
      account: myaccount
      user: myuser
      password: mypassword
      warehouse: compute_wh
      database: analytics
      schema: dbt_dev
EOF

# Run dbt project through SQLMesh
sqlmesh plan
sqlmesh apply

# Convert dbt project to SQLMesh format (optional)
sqlmesh dbt convert

Code Evidence

# File: pyproject.toml:108
dbt = ["dbt-core<2"]

# File: pyproject.toml:64-67
# Known issue with dbt-snowflake 1.10.1
# dbt-snowflake==1.10.1 depends on InvalidCatalogIntegrationConfigError
# which only exists in dbt-adapters>=1.16.6, but the package metadata
# declares dbt-adapters<1.16.0 as valid, causing pip to fail
"dbt-snowflake!=1.10.1",

# File: sqlmesh/dbt/profile.py:72
def load_profile(profile_name: str) -> t.Dict[str, t.Any]:
    """Load dbt profile from profiles.yml."""
    profiles_dir = os.environ.get("DBT_PROFILES_DIR", os.path.expanduser("~/.dbt"))
    profile_path = os.path.join(profiles_dir, "profiles.yml")

    with open(profile_path) as f:
        profiles = yaml.safe_load(f)

    return profiles.get(profile_name, {})

# File: sqlmesh/dbt/builtin.py:434
def enable_macro_debugging() -> bool:
    """Check if macro debugging is enabled."""
    return os.environ.get("DBT_MACRO_DEBUGGING", "").lower() in ("1", "true", "t")

# File: sqlmesh/dbt/loader.py:38
# Uses importlib_metadata for Python < 3.12
try:
    from importlib.metadata import entry_points
except ImportError:
    from importlib_metadata import entry_points  # type: ignore

class DbtLoader:
    """Loads dbt projects into SQLMesh context."""

    def load(self, project_dir: str) -> Context:
        """Load dbt project and return SQLMesh context."""
        # ... implementation

Common Errors

Error Message	Cause	Solution
ModuleNotFoundError: No module named 'dbt'	dbt-core not installed	Install with pip install "sqlmesh[dbt]"
Could not find profile	profiles.yml missing or misconfigured	Create ~/.dbt/profiles.yml with valid profile
InvalidCatalogIntegrationConfigError	Using dbt-snowflake 1.10.1	Upgrade/downgrade: pip install "dbt-snowflake!=1.10.1"
Adapter not found	Engine-specific dbt adapter not installed	Install adapter: pip install dbt-{engine}
Macro compilation failed	Unsupported Jinja syntax	Check macro for SQLMesh compatibility
importlib_metadata not found	Python < 3.12 missing dependency	Install pip install importlib-metadata

Compatibility Notes

dbt-core version 1.x supported (v2 not yet supported)
dbt-snowflake 1.10.1 must be excluded due to dependency resolution bug
dbt-duckdb requires >= 1.7.1 for latest features
All standard dbt materializations supported (table, view, incremental, ephemeral)
Incremental strategies supported: delete+insert, merge, insert_overwrite
SCD Type 2 snapshots fully supported with timestamp and check strategies
Jinja templating and macros converted to SQLMesh equivalents
dbt tests integrated into SQLMesh test framework
Package dependencies resolved using dbt's packages.yml
profiles.yml used for connection configuration (standard dbt)
DBT_PROFILES_DIR environment variable respected
Macro debugging available via DBT_MACRO_DEBUGGING flag
importlib_metadata required for Python < 3.12 package discovery
Full support for dbt sources, seeds, and snapshots
Cross-database compatibility via SQLMesh's multi-dialect support

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment