Implementation:Mage ai Mage ai API Source

Knowledge Sources	Mage Integrations Mage Docs
Domains	Data_Integration, HTTP_API, Source_Connector
Last Updated	2026-02-09 00:00 GMT

Overview

Concrete tool for extracting data from generic HTTP API endpoints provided by the Mage integrations source connector framework.

Description

The Api source connector extends the base Source class to implement data extraction from arbitrary HTTP API endpoints. It supports both GET and POST request methods, configurable headers, query parameters, and request payloads. The connector auto-detects the response content type using MIME-type inspection via the python-magic library and handles multiple formats: JSON responses, CSV, TSV, gzipped CSV, Excel (XLSX), and Google Sheets exported URLs. For JSON responses, an optional response_parser path can be configured to extract nested data using dot notation. When the JSON response contains arrays of non-dict values, the connector auto-generates column names (col0, col1, etc.) or uses explicitly configured column names. The HTTP session is configured with aggressive retry logic (up to 100 retries) and a 12-second timeout. CSV and TSV parsing is handled via the Polars library for performance, with configurable separator and header options. Discovery infers the schema by making an actual API request and analyzing the response data using pandas type inference. The replication method is full-table, and SSL verification is disabled for all requests.

Usage

Use this source connector when building a Mage data pipeline that needs to extract data from a REST API or publicly accessible data file URL. Configure with url as the primary required parameter, plus optional method, headers, query, payload, and response parsing settings.

Code Reference

Source Location

Repository: mage-ai
File: mage_integrations/mage_integrations/sources/api/__init__.py
Lines: 1-270

Signature

class Api(Source):
    @property
    def http_method(self):
        ...
    def discover(self, streams: List[str] = None) -> Catalog:
        ...
    def load_data(self, *args, **kwargs) -> Generator[List[Dict], None, None]:
        ...
    def test_connection(self):
        ...

Import

from mage_integrations.sources.api import Api

I/O Contract

Inputs

Name	Type	Required	Description
config	dict	Yes	Configuration dictionary with URL and request settings
catalog	Catalog	No	Singer catalog specifying streams to extract
state	dict	No	Previous sync state for incremental extraction

Configuration Parameters

Name	Type	Required	Description
url	str	Yes	The HTTP URL to fetch data from
method	str	No	HTTP method: GET (default) or POST
headers	dict	No	Additional HTTP headers to include in the request
query	dict	No	Query parameters to append to the URL
payload	str	No	Request body payload for POST requests
response_parser	str	No	Dot-notation path to extract nested data from JSON responses
columns	list	No	Column names for non-dict JSON array responses
separator	str	No	CSV column separator (defaults to comma)
has_header	bool	No	Whether CSV/TSV files have a header row (defaults to False)

Outputs

Name	Type	Description
catalog	Catalog	Discovered stream named 'api' with schema inferred from response data (from discover())
records	Generator[List[Dict]]	Batches of records parsed from the API response (from load_data())

Usage Examples

from mage_integrations.sources.api import Api

config = {
    "url": "https://api.example.com/v1/data",
    "method": "GET",
    "headers": {
        "Authorization": "Bearer your_token_here",
    },
    "response_parser": "data.results",
}

source = Api(config=config)

# Discover available streams
catalog = source.discover()

# Test connection
source.test_connection()

Related Pages

Implements Principle

Principle:Mage_ai_Mage_ai_Source_Lifecycle_Orchestration

Requires Environment

Environment:Mage_ai_Mage_ai_Python_3_9_Runtime

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment