Implementation:TobikoData Sqlmesh Pyproject Toml
| Knowledge Sources | |
|---|---|
| Domains | Build Configuration, Packaging, Dependencies |
| Last Updated | 2026-02-07 20:00 GMT |
Overview
The pyproject.toml file defines SQLMesh's build configuration, dependencies, optional extras, testing setup, and code quality rules.
Description
This PEP 518 compliant configuration file specifies SQLMesh as a next-generation data transformation framework requiring Python >= 3.9. It declares core dependencies including sqlglot[rs]~=28.10.0, pydantic>=2.0.0, duckdb>=0.10.0, and pandas<3.0.0. The file defines 16+ optional dependency groups for different SQL engines (Snowflake, BigQuery, Databricks, etc.), development tools, and integrations (dbt, GitHub CI/CD, Slack).
The configuration includes extensive mypy type checking rules, pytest markers for test organization (fast/slow/docker/remote/engine-specific), and ruff linting rules that ban module-level imports of duckdb, numpy, and pandas to prevent import-time side effects.
Usage
This file is automatically read by pip, setuptools, and other Python packaging tools. Developers reference it to install SQLMesh with specific extras, while CI/CD systems use it for consistent dependency management.
Code Reference
Source Location
- Repository: TobikoData_Sqlmesh
- File: pyproject.toml
Project Metadata
[project]
name = "sqlmesh"
requires-python = ">= 3.9"
description = "Next-generation data transformation framework"
Entry Points
[project.scripts]
sqlmesh = "sqlmesh.cli.main:cli"
sqlmesh_dbt = "sqlmesh_dbt.cli:dbt"
sqlmesh_cicd = "sqlmesh.cicd.bot:bot"
sqlmesh_lsp = "sqlmesh.lsp.main:main"
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| Python version | string | Yes | Python >= 3.9, < 3.13 (3.13+ not yet supported) |
| pip install extras | list[string] | No | Optional dependency groups like [snowflake], [dev] |
Outputs
| Name | Type | Description |
|---|---|---|
| sqlmesh command | CLI | Main SQLMesh command-line interface |
| sqlmesh_dbt command | CLI | dbt integration CLI |
| sqlmesh_cicd command | CLI | GitHub CI/CD bot |
| sqlmesh_lsp command | CLI | Language Server Protocol implementation |
| Package metadata | dict | Version, dependencies, classifiers for PyPI |
Core Dependencies
Essential Packages
| Package | Version Constraint | Purpose |
|---|---|---|
| sqlglot[rs] | ~=28.10.0 | SQL parsing and transpilation (Rust-accelerated) |
| pydantic | >=2.0.0 | Data validation and settings management |
| duckdb | >=0.10.0,!=0.10.3 | Default SQL engine |
| pandas | <3.0.0 | DataFrame operations |
| click | (latest) | CLI framework |
| jinja2 | (latest) | Template rendering |
| croniter | (latest) | Cron expression parsing |
| rich[jupyter] | (latest) | Terminal and notebook formatting |
| tenacity | (latest) | Retry logic |
Optional Dependency Groups
Cloud Data Warehouses
[project.optional-dependencies]
snowflake = [
"cryptography<46.0.0",
"snowflake-connector-python[pandas,secure-local-storage]",
"snowflake-snowpark-python",
]
bigquery = [
"google-cloud-bigquery[pandas]",
"google-cloud-bigquery-storage"
]
databricks = ["databricks-sql-connector[pyarrow]"]
redshift = ["redshift_connector"]
athena = ["PyAthena[Pandas]"]
Database Engines
postgres = ["psycopg2"]
mysql = ["pymysql"]
mssql = ["pymssql"]
mssql-odbc = ["pyodbc>=5.0.0"]
trino = ["trino"]
clickhouse = ["clickhouse-connect"]
risingwave = ["psycopg2"]
Integrations
dbt = ["dbt-core<2"]
github = ["PyGithub>=2.6.0"]
slack = ["slack_sdk"]
mwaa = ["boto3"]
dlt = ["dlt"]
web = [
"fastapi==0.115.5",
"uvicorn[standard]==0.22.0",
"sse-starlette>=0.2.2",
"pyarrow",
]
Development Tools
dev = [
"pytest",
"pytest-xdist",
"mypy~=1.13.0",
"ruff~=0.11.0",
"pre-commit",
# All engine connectors for testing
# All dbt adapters
]
Testing Configuration
Pytest Markers
| Marker | Category | Description |
|---|---|---|
| fast | Type | Fast tests (automatically applied if no type markers) |
| slow | Type | Slow tests involving local DB interactions |
| docker | Type | Tests requiring Docker containers |
| remote | Type | Tests requiring remote database connections |
| cicdonly | Type | Tests only run on CI/CD |
| isolated | Type | Tests needing sequential execution (uses fork) |
| engine | Domain | Test all engine adapters |
| snowflake | Domain | Snowflake-specific tests |
| bigquery | Domain | BigQuery-specific tests |
| databricks | Domain | Databricks-specific tests |
| duckdb | Domain | DuckDB-specific tests |
| cli | Domain | CLI tests |
| dbt | Domain | dbt adapter tests |
| github | Domain | GitHub CI/CD bot tests |
| web | Domain | Web UI tests |
Pytest Settings
[tool.pytest.ini_options]
markers = [
"fast: fast tests (automatically applied if no type markers)",
"engine: test all engine adapters",
# 16+ engine-specific markers
]
addopts = "-n 0 --dist=loadgroup"
log_cli = false
reruns_delay = 10
Code Quality Tools
MyPy Type Checking
[tool.mypy]
plugins = "pydantic.mypy"
no_implicit_optional = true
disallow_untyped_defs = true
[[tool.mypy.overrides]]
module = [
"tests.*",
"sqlmesh.migrations.*"
]
disallow_untyped_defs = false
Ruff Linting
[tool.ruff]
line-length = 100
[tool.ruff.lint]
select = ["F401", "RET505", "T100"]
extend-select = ["TID"]
[tool.ruff.lint.flake8-tidy-imports]
banned-module-level-imports = [
"duckdb",
"numpy",
"pandas",
]
Key Rule: duckdb, numpy, and pandas are banned at module level to prevent import-time side effects and improve load times.
LSP Import Isolation
[tool.ruff.lint.flake8-tidy-imports.banned-api]
"sqlmesh.lsp".msg = "Only files within sqlmesh/lsp can import from sqlmesh.lsp"
Usage Examples
Basic Installation
# Core SQLMesh only
pip install sqlmesh
# With Snowflake support
pip install "sqlmesh[snowflake]"
# With BigQuery and dbt
pip install "sqlmesh[bigquery,dbt]"
# Development installation with all engines
pip install "sqlmesh[dev]"
make install-dev
Running Tests by Marker
# Fast tests only (default for development)
pytest -m fast
# Slow tests (local database interactions)
pytest -m slow
# Snowflake tests (requires credentials)
pytest -m snowflake
# All tests except docker and slow
pytest -m "not slow and not docker"
# Specific engine with verbose output
pytest -m duckdb -v
Building Package
# Build wheel and sdist
make package
# Or using build directly
python -m build
Type Checking
# Run mypy on entire codebase
mypy sqlmesh/
# Type check specific module
mypy sqlmesh/core/context.py
Linting
# Run all style checks
make style
# Just ruff
ruff check sqlmesh/
# Auto-fix issues
ruff check --fix sqlmesh/
Build System
Setuptools Configuration
[build-system]
requires = ["setuptools >= 61.0", "setuptools_scm"]
build-backend = "setuptools.build_meta"
[tool.setuptools_scm]
version_file = "sqlmesh/_version.py"
fallback_version = "0.0.0"
local_scheme = "no-local-version"
[tool.setuptools.packages.find]
include = ["sqlmesh", "sqlmesh.*", "sqlmesh_dbt", "sqlmesh_dbt.*", "web*"]
[tool.setuptools.package-data]
web = ["client/dist/**"]
"*" = ["py.typed"]
Version is automatically determined from git tags using setuptools_scm.
Related Pages
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment