Principle:Mage ai Mage ai Source Configuration
| Knowledge Sources | |
|---|---|
| Domains | Data_Integration, ETL |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A configuration parsing pattern that initializes Singer-compatible source connectors from CLI arguments, JSON config, state files, and YAML settings.
Description
Source Configuration is the foundational step in any Singer tap (data extraction) workflow. It parses command-line arguments to load database/API connection credentials, sync state for incremental replication, catalog definitions for stream selection, and operational mode flags (discover, sync, test_connection). The pattern supports multiple input formats: file paths to JSON/YAML configs, inline JSON strings, and YAML settings files that bundle config and catalog together.
Usage
Use this principle at the very start of any data extraction pipeline. It must execute before schema discovery, data loading, or any other tap operation. It applies whenever a source connector needs to be initialized with connection credentials and sync parameters.
Theoretical Basis
The Singer specification defines a standard CLI interface for taps:
- -c/--config - Path to JSON config file with connection credentials
- -s/--state - Path to JSON state file with bookmark values from previous syncs
- --catalog - Path to catalog file defining stream schemas and selection
- -d/--discover - Flag to run in discovery mode
The configuration phase resolves all inputs into a normalized set of instance attributes (config dict, state dict, catalog object, query dict, mode flags) that downstream steps consume. This decouples input format handling from business logic.