Heuristic:Mage ai Mage ai HTTP Error Classification
| Knowledge Sources | |
|---|---|
| Domains | Debugging, Data_Integration |
| Last Updated | 2026-02-09 07:00 GMT |
Overview
Map HTTP status codes to semantic exception types with fallback classification for unknown codes.
Description
The base HTTP client maps known HTTP status codes (400, 401, 403, 404, 405, 409, 429, 500, 503) to specific exception classes (BadRequestError, AuthenticationError, RateLimitError, etc.). Unknown status codes in the 4xx range fall back to `Server4xxError`, 5xx to `Server5xxError`, and anything else to the generic `HttpError`. This enables downstream code to catch specific error types for targeted handling (e.g., retry on RateLimitError, fail fast on AuthenticationError).
Usage
Apply this heuristic when:
- Building a new API source connector: Extend the base Client class and rely on the built-in error classification for HTTP responses.
- Implementing retry logic: Catch `RateLimitError` (429) and `ServiceUnavailableError` (503) for retries; do not retry `AuthenticationError` (401) or `ForbiddenError` (403).
- Debugging API extraction failures: The exception message includes both the HTTP status code and the API's error message for diagnostics.
The Insight (Rule of Thumb)
- Action: Use `raise_for_error(response)` from the base HTTP client rather than implementing custom error handling per connector.
- Retryable errors: 429 (RateLimitError), 500 (InternalServiceError), 503 (ServiceUnavailableError).
- Non-retryable errors: 400 (BadRequestError), 401 (AuthenticationError), 403 (ForbiddenError), 404 (NotFoundError).
- Trade-off: The fallback classification means any unknown 4xx/5xx gets a generic error type, which may miss edge cases specific to certain APIs.
Reasoning
Data integration pipelines interact with dozens of different APIs, each with their own error semantics. A centralized error classification ensures consistent handling across all connectors. The fallback to generic 4xx/5xx classes prevents crashes on unexpected status codes while still preserving the client/server distinction for retry decisions.
The 429 (Rate Limit) classification is particularly important for API connectors that use the `@utils.ratelimit(100, 60)` decorator, as it allows backoff-and-retry logic to distinguish "slow down" from "you are unauthorized".
Code Evidence
Status code mapping from `sources/http/client.py:61-98`:
STATUS_CODE_EXCEPTION_MAPPING = {
400: {"raise_exception": BadRequestError, "message": "..."},
401: {"raise_exception": AuthenticationError, "message": "..."},
403: {"raise_exception": ForbiddenError, "message": "..."},
404: {"raise_exception": NotFoundError, "message": "..."},
405: {"raise_exception": MethodNotAllowedError, "message": "..."},
409: {"raise_exception": NotProcessedError, "message": "..."},
429: {"raise_exception": RateLimitError, "message": "..."},
500: {"raise_exception": InternalServiceError, "message": "..."},
503: {"raise_exception": ServiceUnavailableError, "message": "..."},
}
Fallback classification from `sources/http/client.py:101-115`:
def get_exception_for_status_code(status_code):
exception = STATUS_CODE_EXCEPTION_MAPPING.get(status_code, {}).get(
"raise_exception")
if not exception:
if status_code > 400 and status_code < 500:
exception = Server4xxError
elif status_code > 500:
exception = Server5xxError
else:
exception = HttpError
return exception