Principle:Mage ai Mage ai Source Connector Scaffolding
| Knowledge Sources | |
|---|---|
| Domains | Data_Integration, Software_Architecture |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A standardized directory structure pattern for creating new Singer-compatible source connectors within the Mage integrations framework.
Description
Source Connector Scaffolding defines the file and directory layout that every source connector in the Mage integrations framework must follow. This convention ensures consistency across 40+ connectors and enables the framework's automatic schema loading, template serving, and entry point resolution. The pattern varies by connector type: SQL connectors extend SQLSource and rely on information_schema for discovery, while API connectors extend Source/Client and use static JSON schema files.
Usage
Follow this pattern when creating any new source connector. The directory structure must be created before implementing the connector class.
Theoretical Basis
Standard Directory Layout
sources/{source_name}/
__init__.py # Main connector class (extends Source or SQLSource)
README.md # Configuration documentation
schemas/ # JSON Schema files (one per stream) - API sources only
stream_name.json
templates/ # Config template files
config.json
Extended Layout (API connectors with multiple streams)
sources/{source_name}/
__init__.py # Main connector class
client.py # HTTP client (extends Client)
README.md
schemas/
stream1.json
stream2.json
streams/
__init__.py
base.py # Base stream class
stream1.py # Per-stream extraction logic
templates/
config.json
Connector Type Selection
- SQL databases -> Extend SQLSource, override build_connection()
- REST APIs -> Extend Source + Client, override load_data() and get_headers()
- Simple/file sources -> Extend Source, override load_data()