Heuristic:TobikoData Sqlmesh Forward Only Safety
| Knowledge Sources | |
|---|---|
| Domains | Deployment, Safety |
| Last Updated | 2026-02-07 21:00 GMT |
Overview
Forward-only deployment prevents expensive or dangerous backfilling of historical data when modifying incremental models.
Description
When modifying incremental models, SQLMesh normally requires reprocessing historical data to ensure consistency. However, for certain additive changes (like adding columns) or when historical reprocessing is prohibitively expensive, the forward_only flag allows changes to take effect only for future data. This is classified as SnapshotChangeCategory.FORWARD_ONLY and requires no rebuilding of existing snapshots. The system provides granular control through allow_destructive_models and allow_additive_models configuration options.
Usage
Use forward-only deployment when:
- Adding new columns to incremental models (additive changes)
- Historical data reprocessing would be too expensive in time or compute
- Changes don't affect the meaning of existing historical data
- You need to deploy quickly without triggering full backfills
Never use forward-only for:
- Breaking schema changes (column type changes, removals)
- Logic changes that would make historical data inconsistent
- Changes affecting downstream model dependencies (unless explicitly allowing destructive changes)
The Insight (Rule of Thumb)
- Action: Use forward_only=True for additive changes to incremental models, combined with effective_from for production timing control
- Value: Set in plan creation or model definition; use effective_from to specify when changes take effect
- Trade-off: Faster deployment and no historical reprocessing, but historical data lacks new columns/logic
Reasoning
Incremental models process data in time-based chunks (intervals). When a model changes, SQLMesh's default behavior is to reprocess all historical intervals to ensure consistency. This can take hours or days for large datasets. Forward-only deployment recognizes that certain changes (especially additive ones) don't require historical consistency - they can simply apply to new data going forward.
The system categorizes changes into five types: 1. BREAKING: Requires rebuild of modified snapshot and downstream dependencies 2. NON_BREAKING: Requires rebuild of only the modified snapshot 3. FORWARD_ONLY: Requires no rebuilding 4. INDIRECT_BREAKING: Caused indirectly and is breaking 5. INDIRECT_NON_BREAKING: Caused indirectly by a non-breaking change
Forward-only changes get special handling in the plan builder through the _forward_only_preview_needed optimization, which avoids expensive preview computation when it's not necessary.
Code Evidence
# sqlmesh/core/snapshot/definition.py:72-76
BREAKING: The change requires that snapshot modified and downstream dependencies be rebuilt
NON_BREAKING: The change requires that only the snapshot modified be rebuilt
FORWARD_ONLY: The change requires no rebuilding
INDIRECT_BREAKING: The change was caused indirectly and is breaking.
INDIRECT_NON_BREAKING: The change was caused indirectly by a non-breaking change.
# sqlmesh/core/plan/builder.py:78
# forward_only flag usage in PlanBuilder
forward_only: bool = False
# sqlmesh/core/plan/builder.py:888-901
# PlanBuilder has _forward_only_preview_needed cached property
# to optimize preview computation for forward-only changes
Example usage in SQL model:
MODEL (
name my_schema.my_model,
kind INCREMENTAL_BY_TIME_RANGE(time_column),
forward_only true,
effective_from '2026-02-01'
);
SELECT
existing_column,
new_column -- This will only exist in data from 2026-02-01 forward
FROM source_table