Principle:TobikoData Sqlmesh Project Configuration
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, DevOps, Build_Systems |
| Last Updated | 2026-02-07 00:00 GMT |
Overview
Project configuration encompasses the declarative specification files that define a SQLMesh project's build settings, dependency management, CI/CD pipeline definitions, and API contracts, enabling reproducible builds, automated testing, and consistent toolchain behavior across development environments.
Description
A SQLMesh project relies on several configuration artifacts that sit outside the core Python package but are essential for project operation. These include the Python project definition (`pyproject.toml`) which specifies package metadata, dependencies, and build system configuration; CI/CD pipeline definitions (CircleCI configuration) that automate testing across multiple database engines and Python versions; dependency lockfiles (`pnpm-lock.yaml`) that ensure deterministic frontend builds; and OpenAPI specifications that define the contract between the web server and its clients.
The `pyproject.toml` serves as the single source of truth for the Python package, defining required and optional dependencies (organized by database engine), entry points for CLI commands (`sqlmesh` and `sqlmesh_cicd`), linting and formatting rules, and test runner configuration. The CircleCI configuration orchestrates a complex matrix of integration tests across multiple database engines (DuckDB, PostgreSQL, Snowflake, BigQuery, Databricks, etc.) and Python versions.
The OpenAPI specifications are generated from the FastAPI server and consumed by both the web client (via Orval code generation) and the VSCode extension, ensuring type-safe API interactions. These specs serve as the contract between backend and frontend components.
Usage
Project configuration files are consulted during initial project setup, CI/CD pipeline execution, dependency updates, and release processes. They are modified when adding new database engine support, updating dependencies, changing build processes, or extending the API surface. These files are critical infrastructure that should be updated carefully with proper review.
Theoretical Basis
Project configuration follows established software engineering practices:
Declarative Configuration:
- Single `pyproject.toml` replaces multiple config files (setup.py, setup.cfg, MANIFEST.in)
- CI/CD pipelines defined as code (YAML) for reproducibility
- Dependency pinning via lockfiles for deterministic builds
- OpenAPI specs as machine-readable API contracts
Dependency Management:
- Core dependencies separated from optional engine-specific dependencies
- Frontend dependencies managed via pnpm with workspace support
- Lockfiles committed to source control for reproducible installs
- Version constraints specified using semantic versioning ranges
Build System Organization:
- Python packaging via setuptools with pyproject.toml
- Frontend builds via Vite with TypeScript compilation
- API client generation via Orval from OpenAPI specs
- Documentation generation via MkDocs with Material theme
CI/CD Pipeline Design:
- Parallel job execution across database engine matrix
- Conditional workflows triggered by file change patterns
- Shared configuration via YAML anchors and continuation orbs
- Artifact caching for dependency installation acceleration