Principle:Mage ai Mage ai Incremental Sync State
| Knowledge Sources | |
|---|---|
| Domains | Data_Integration, ETL, Incremental_Processing |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A bookmark-based state persistence mechanism that enables incremental data extraction by tracking the last synced position per stream.
Description
Incremental Sync State is the Singer protocol's mechanism for avoiding full re-extraction on every run. After each batch of records is extracted, the source emits a STATE message containing bookmark values (typically the maximum value of a replication key like updated_at or an auto-incrementing ID). On subsequent runs, the tap loads the previous state and uses these bookmarks as WHERE clause filters, extracting only records newer than the last checkpoint.
Usage
Use this principle for any stream configured with INCREMENTAL or LOG_BASED replication methods. It is critical for large datasets where full table extraction would be impractical. Not applicable to FULL_TABLE replication.
Theoretical Basis
The state lifecycle follows this pattern:
- Load: Previous state is loaded from a JSON file or inline JSON at startup
- Filter: Bookmark values are used in WHERE clauses (e.g., WHERE updated_at >= bookmark_value)
- Track: During extraction, max bookmark values are tracked per record
- Emit: STATE messages are written to stdout after each record batch (if sorted) or after all records (if unsorted)
- Persist: Downstream target saves the state for the next run
State structure: {"bookmarks": {"stream_id": {"replication_key": "value"}}}