Implementation:Datahub project Datahub Avro Schema To Mce

Knowledge Sources	Datahub_project_Datahub
Domains	Schema_Conversion, Metadata_Ingestion
Last Updated	2026-02-10 00:00 GMT

Overview

Description

avro_schema_to_mce.py is a Python CLI script that converts Avro schema files into DataHub SchemaMetadata aspects and emits them as MetadataChangeProposal events. It uses the Click framework for CLI argument parsing and supports two output modes: writing to a JSON file via SynchronizedFileEmitter and/or emitting directly to a DataHub server via the default graph client.

The script performs the following steps:

1. Reads and parses an Avro schema file using the avro library.
2. Computes the schema's MD5 fingerprint from its canonical form.
3. Validates that the schema is a RecordSchema.
4. Constructs a dataset URN from the platform and schema namespace/name.
5. Converts Avro fields to DataHub SchemaField objects using AvroToMceSchemaConverter.
6. Builds a SchemaMetadataClass with the fields, hash, and raw schema.
7. Emits the result to a file and/or DataHub server.

Usage

Used as a standalone CLI tool within the SchemaTron toolchain for converting Avro schemas to DataHub metadata. Can be invoked directly or as a Click command.

Code Reference

Source Location

metadata-integration/java/datahub-schematron/cli/scripts/avro_schema_to_mce.py

Signature

@click.command(name="avro2datahub")
@click.option("--input-file", "-i", type=click.Path(exists=True), required=True)
@click.option("--platform", type=str, required=True)
@click.option("--output-file", "-o", type=click.Path(), default="metadata.py.json")
@click.option("--to-file", "-f", is_flag=True, default=True)
@click.option("--to-server", "-s", is_flag=True, default=False)
def generate_schema_file_from_avro_schema(
    input_file: str, platform: str, output_file: str, to_file: bool, to_server: bool
) -> None

def get_schema_hash(schema) -> str

Import

from datahub.ingestion.extractor.schema_util import AvroToMceSchemaConverter
from datahub.emitter.synchronized_file_emitter import SynchronizedFileEmitter
from datahub.emitter.mce_builder import make_data_platform_urn, make_dataset_urn
from datahub.emitter.mcp import MetadataChangeProposalWrapper
import datahub.metadata.schema_classes as models

I/O Contract

Inputs

Parameter	Type	Description
--input-file / -i	str (path)	Path to the Avro schema file (must exist)
--platform	str	Data platform name (e.g., "kafka", "hive")
--output-file / -o	str (path)	Output JSON file path (default: metadata.py.json)
--to-file / -f	bool (flag)	Write output to file (default: True)
--to-server / -s	bool (flag)	Emit to DataHub server (default: False)

Outputs

Output	Description
JSON file	A file containing the serialized MetadataChangeProposal with SchemaMetadata aspect
DataHub server	Direct emission to the configured DataHub GMS server (when --to-server is used)

Usage Examples

# Convert Avro schema to file
python avro_schema_to_mce.py -i schema.avsc --platform kafka -o output.json

# Convert and emit to server
python avro_schema_to_mce.py -i schema.avsc --platform hive --to-server

# Both file and server
python avro_schema_to_mce.py -i schema.avsc --platform kafka -o output.json --to-server

Related Pages

• Datahub_project_Datahub_SchemaTron_CLI -- Java counterpart for Avro schema conversion
• Datahub_project_Datahub_MetadataChangeProposalWrapper_Java -- Java equivalent of the MCP wrapper used here