Principle:Mage ai Mage ai Schema Validation
| Knowledge Sources | |
|---|---|
| Domains | Data_Integration, Data_Quality, Schema_Management |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
A schema registration and validation mechanism that creates JSON Schema validators for incoming streams to ensure record data conformity before loading into the target.
Description
Schema Validation ensures data quality at the destination by registering stream schemas from incoming SCHEMA messages and creating Draft4Validator instances for each stream. When records arrive, they are validated against the registered schema to catch type mismatches, missing required fields, and invalid values before they reach the export layer. The Mage framework extends standard validation with internal columns, stream override settings, and configurable disable_column_type_check per stream.
Usage
Applied automatically when SCHEMA messages arrive at the destination. Must occur before any RECORD messages for a given stream.
Theoretical Basis
Schema registration stores per-stream:
- schemas[stream] - JSON Schema dict with properties
- validators[stream] - jsonschema.Draft4Validator instance
- key_properties[stream] - Primary key columns
- bookmark_properties[stream] - Replication key columns
- replication_methods[stream] - FULL_TABLE or INCREMENTAL
- partition_keys[stream] - Partition columns for target storage
- unique_constraints[stream] - Unique constraint columns
- unique_conflict_methods[stream] - How to handle upsert conflicts