Principle:Mage ai Mage ai HTTP Client Request
| Knowledge Sources | |
|---|---|
| Domains | Data_Integration, HTTP, API |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A rate-limited HTTP client pattern that provides authenticated API requests with configurable timeouts and structured error handling for Singer API source connectors.
Description
HTTP Client Request encapsulates the common needs of API-based data extraction: sending authenticated HTTP requests with rate limiting, configurable timeouts, and structured error handling mapped to specific HTTP status codes. The pattern uses a base class that subclasses override to inject authentication (headers, API keys, OAuth tokens) and customize request parameters. Rate limiting is enforced via a decorator pattern (100 requests per 60 seconds by default).
Usage
Use this principle when building source connectors for REST APIs, SaaS platforms, or any HTTP-based data source. Not applicable to SQL database sources.
Theoretical Basis
The HTTP client pattern follows these design principles:
- Rate limiting: Decorator-based throttling (e.g., 100 requests/60 seconds) prevents API quota exhaustion
- Authentication injection: Subclasses override get_headers() to add API keys, Bearer tokens, or Basic Auth
- Structured error mapping: HTTP status codes map to specific exception classes (400 -> BadRequestError, 401 -> AuthenticationError, 429 -> RateLimitError, 500 -> InternalServiceError)
- Configurable timeout: Default 300 seconds, overridable via config['request_timeout']