Principle:Mage ai Mage ai Singer Message Emission
| Knowledge Sources | |
|---|---|
| Domains | Data_Integration, ETL, Serialization |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A stdout-based message serialization protocol that emits SCHEMA, RECORD, and STATE messages as JSON lines for downstream consumption by Singer targets.
Description
Singer Message Emission is the output protocol for source connectors. After loading data, the source must serialize extracted records into the Singer wire format: JSON lines written to stdout. Three message types are emitted: SCHEMA (stream metadata including JSON Schema, key_properties, bookmark_properties), RECORD (individual data records), and STATE (bookmark checkpoints). The Mage framework extends the standard Singer protocol with additional fields: replication_method, partition_keys, unique_conflict_method, unique_constraints, and disable_column_type_check.
Usage
This principle applies to every source connector during the sync phase. SCHEMA is emitted once per stream before records. RECORD is emitted for each extracted row. STATE is emitted periodically for incremental bookmarking.
Theoretical Basis
Singer message format (JSON lines on stdout):
- SCHEMA:
{"type": "SCHEMA", "stream": "...", "schema": {...}, "key_properties": [...]} - RECORD:
{"type": "RECORD", "stream": "...", "record": {...}} - STATE:
{"type": "STATE", "value": {"bookmarks": {...}}}
Each line is serialized with simplejson (supports Decimal, NaN handling) and flushed immediately.