Implementation:Datahub project Datahub Mce Diff Script

Knowledge Sources	Datahub_project_Datahub
Domains	Schema_Testing
Last Updated	2026-02-10 00:00 GMT

Overview

A Python CLI tool that computes structural diffs between two Metadata Change Event (MCE) JSON files, supporting golden diff comparison and update workflows for schema conversion regression testing.

Description

The mce_diff.py script is part of the datahub-schematron CLI tooling and provides recursive dictionary and list comparison tailored to DataHub MCE/MCP JSON structures. It is used to validate that schema converter outputs remain consistent across changes.

Core functions:

• diff_dicts(dict1, dict2) -- Recursively compares two dictionaries, returning a result with keys: added, removed, modified, and modified_details. Includes special-case handling: if a dict contains a nullable key, it delegates to diff_schema_field(); if it contains a hash key, it delegates to diff_schema_metadata().
• diff_lists(list1, list2) -- Compares two lists element by element. When lists differ in length, it attempts to align elements by fieldPath key if present, then reports added/removed/modified elements.
• diff_schema_field(field1_dict, field2_dict) -- Specialized diff for SchemaFieldClass objects, comparing fieldPath, type, description, and nullable.
• diff_schema_metadata(schema1_dict, schema2_dict) -- Specialized diff that ignores volatile fields (created, modified, hash, platformSchema, lastModified) before comparison.
• process_single_element(element) -- Extracts (entityUrn, aspectName, aspect_json) tuples from MCE elements.
• compute_diff() -- The Click CLI entry point that loads two MCE JSON files, processes them into entity-aspect dictionaries, and either asserts no diff, compares against a golden diff file, or updates the golden diff file.

Usage

Use this script during development and CI to verify that changes to schema converters (e.g., AvroSchemaConverter) produce expected output. The golden diff workflow allows intentional schema changes to be recorded and validated against future runs.

Code Reference

Source Location

• Repository: Datahub_project_Datahub
• File: metadata-integration/java/datahub-schematron/cli/scripts/mce_diff.py

Signature

def diff_lists(list1, list2) -> dict:
def diff_dicts(dict1, dict2, identifier=None) -> dict:
def diff_schema_field(field1_dict, field2_dict) -> dict:
def diff_schema_metadata(schema1_dict, schema2_dict) -> dict:
def is_empty_diff(diff_dict) -> bool:
def format_diff(diff_dict) -> Any:
def EMPTY_DIFF() -> dict:
def process_single_element(element) -> Tuple[str, str, Dict[str, Any]]:
def process_element_with_dict(element, global_dict) -> None:

@click.command("compute_diff")
def compute_diff(
    input_file_1: str,
    input_file_2: str,
    golden_diff_file: Optional[str] = None,
    update_golden_diff: bool = False,
) -> None:

Import

from scripts.mce_diff import compute_diff, diff_dicts, diff_lists

I/O Contract

Input	Type	Description
input_file_1	File path	First MCE JSON file (list of entity/aspect elements)
input_file_2	File path	Second MCE JSON file to compare against
--golden-diff-file	File path	Optional golden diff JSON for regression testing
--update-golden-diff	Flag	When set, writes computed diff to the golden diff file

Output	Type	Description
Diff result	dict	Structure with added, removed, modified, modified_details keys
Exit status	int	Assertion failure if diffs are unexpected; success if diffs match golden or are empty
Golden diff file	JSON file	Updated golden diff file when --update-golden-diff is set

Usage Examples

# CLI usage: compare two MCE files expecting no diff
# python mce_diff.py file1.json file2.json

# CLI usage: compare against golden diff
# python mce_diff.py file1.json file2.json --golden-diff-file golden.json

# CLI usage: update golden diff
# python mce_diff.py file1.json file2.json --golden-diff-file golden.json --update-golden-diff

# Programmatic usage
from scripts.mce_diff import diff_dicts
result = diff_dicts(
    {"fieldPath": "name", "type": "string", "nullable": True},
    {"fieldPath": "name", "type": "string", "nullable": False},
)
# result: {"nullable": {"before": True, "after": False, "identifier": "name"}}