Workflow:TobikoData Sqlmesh Dbt project migration
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, dbt, Migration |
| Last Updated | 2026-02-07 21:00 GMT |
Overview
End-to-end process for migrating an existing dbt project to run under SQLMesh, gaining access to virtual environments, plan/apply workflow, and multi-dialect SQL support while preserving existing dbt models and macros.
Description
This workflow enables teams with existing dbt projects to adopt SQLMesh without rewriting their transformations. SQLMesh provides native dbt compatibility by parsing dbt manifests, translating Jinja macros, and mapping dbt materializations to SQLMesh model kinds. Teams can either run their dbt project directly under SQLMesh (hybrid mode) or convert it fully to SQLMesh's native format. The process covers initialization, configuration mapping, model validation, and incremental adoption.
Key outcomes:
- Existing dbt models running under SQLMesh's plan/apply workflow
- Access to virtual environments for safe development
- Multi-dialect SQL transpilation for cross-engine compatibility
- Optional full conversion to SQLMesh native format
Usage
Execute this workflow when your team has an existing dbt project and wants to adopt SQLMesh's advanced features (virtual environments, plan/apply, incremental processing) without a full rewrite. Also suitable for teams wanting to gradually migrate from dbt to SQLMesh.
Execution Steps
Step 1: Project Assessment
Evaluate the existing dbt project structure including models, seeds, sources, tests, snapshots, and macros. Identify dbt packages and dependencies, review the profiles.yml for connection configurations, and assess custom materializations that may require special handling.
Key considerations:
- Inventory all dbt asset types (models, seeds, sources, tests, snapshots, macros)
- Identify external dbt packages and their compatibility
- Note custom materializations and complex Jinja patterns
- Review incremental strategies (delete+insert, merge, insert_overwrite)
Step 2: SQLMesh Initialization for dbt
Initialize SQLMesh within the existing dbt project directory using the dbt template. This configures SQLMesh to read the dbt project structure, parse the dbt manifest, and use the existing profiles.yml for database connections. The initialization creates a SQLMesh configuration that references the dbt project.
Key considerations:
- The dbt project structure remains unchanged
- SQLMesh reads profiles.yml for connection configuration
- The dbt manifest is parsed to discover all project assets
- A SQLMesh config.yaml is created alongside the dbt project
Step 3: Model Discovery and Mapping
SQLMesh's dbt loader discovers all models from the dbt manifest and maps them to SQLMesh's internal representation. dbt materializations are translated to SQLMesh model kinds: table maps to FULL, view maps to VIEW, incremental maps to the appropriate incremental kind, and ephemeral maps to embedded CTEs.
Key considerations:
- Standard dbt materializations map directly to SQLMesh kinds
- Incremental strategies (delete+insert, merge) are preserved
- SCD Type 2 (snapshot) models are supported
- Ephemeral models become embedded subqueries
Step 4: Macro and Jinja Translation
SQLMesh processes dbt Jinja templates and macros, translating them into SQLMesh's macro system. Built-in dbt macros (ref, source, config, is_incremental) are natively supported. Custom macros are translated with their full Jinja context preserved.
Key considerations:
- Built-in dbt macros work out of the box
- Custom macros may require manual review for complex patterns
- Runtime variables (var) are supported through SQLMesh's variable system
- dbt packages and their macros are loaded automatically
Step 5: Validation and Testing
Run the dbt project under SQLMesh to validate that all models parse correctly, dependencies resolve, and transformations produce expected results. Execute existing dbt tests through SQLMesh's test framework to verify compatibility.
Key considerations:
- Parse errors indicate Jinja patterns needing manual adjustment
- Run a plan to verify all models are discovered and categorized
- Execute tests to validate transformation correctness
- Compare output between dbt and SQLMesh for critical models
Step 6: Optional Full Conversion
For teams choosing to fully adopt SQLMesh, use the dbt converter to translate models from Jinja-templated SQL to SQLMesh's native format. This converts Jinja refs to direct model references, dbt config blocks to SQLMesh MODEL blocks, and dbt tests to SQLMesh audit definitions.
Key considerations:
- Conversion is optional; hybrid mode works indefinitely
- Converted models gain access to SQLMesh-specific features
- The converter handles standard patterns automatically
- Complex Jinja logic may need manual conversion