Principle:Mage ai Mage ai Source Lifecycle Orchestration
| Knowledge Sources | |
|---|---|
| Domains | Data_Integration, ETL, Orchestration |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A multi-mode orchestration pattern that routes source connector execution through test_connection, discover, count_records, templates, sample_data, or sync modes based on CLI flags.
Description
Source Lifecycle Orchestration is the top-level control flow for source connectors. Based on parsed CLI flags, it routes execution to the appropriate mode: test_connection validates connectivity, discover outputs the catalog, count_records returns record counts, show_templates outputs config templates, load_sample_data returns a data preview, and the default sync mode runs the full extraction pipeline (discover -> catalog selection -> sync -> write records). The orchestrator handles auto_add_new_fields for schema evolution and filters streams by selected_streams.
Usage
This is the main entry point for all source connectors, called via Source.process() from the main() function.
Theoretical Basis
Mode priority (first match wins):
- test_connection -> Validate connectivity, exit
- load_sample_data -> Load and return sample records as JSON
- discover_mode -> Output catalog to stdout
- count_records -> Return record counts per stream
- show_templates -> Output config templates
- sync (default) -> Full extraction: discover -> catalog -> sync -> write
The sync flow: process() -> sync(catalog) -> for each stream: process_stream(write SCHEMA) -> sync_stream(load_data -> write_records)