Principle:Datahub project Datahub Event Filtering

Metadata

Field	Value
Principle ID	P-DHACT-003
Title	Event Filtering
Category	Event-Driven Automation
Status	Active
Last Updated	2026-02-10
Repository	Datahub_project_Datahub
Knowledge Sources	GitHub - datahub-project/datahub, DataHub Documentation
Domains	Event_Processing, Automation, Metadata_Management

Overview

The mechanism for selectively processing metadata change events based on event type and content matching rules. Event filtering allows action pipelines to react only to relevant events, preventing unnecessary processing and enabling precise targeting of specific metadata changes.

Description

Event filtering allows action pipelines to react only to relevant events. The filtering mechanism operates at two levels:

Event Type Matching

Every event in the DataHub Actions framework carries a type string. The filter matches against this type, supporting both single values and lists. The supported event types are:

EntityChangeEvent_v1: High-level semantic events describing entity-level changes (e.g., tag added, owner changed, documentation updated)
MetadataChangeLogEvent_v1: Low-level aspect-level change events from the versioned metadata change log
MetadataChangeLogEvent_v1 (timeseries): Low-level change events from the timeseries metadata change log

When a list of event types is provided, the filter uses ANY semantics -- the event passes if it matches any one of the listed types.

Event Body Matching

Beyond type matching, the filter supports optional nested JSON body matching with the following semantics:

Dictionary matching: Recursive key-by-key comparison. All specified keys must match (ALL semantics). If a value in the match dictionary is itself a dict, matching recurses into it. If the target value is a JSON string, it is automatically parsed before comparison.
List matching: When the match specification is a list, it uses ANY semantics -- the event body field must equal at least one value in the list.
Scalar matching: Direct equality comparison for primitive values.

This two-level approach (type + body) enables both coarse-grained filtering (react to all entity changes) and fine-grained targeting (react only when a specific tag is added to a dataset).

Usage

Use this principle when an action pipeline should only process specific types of metadata changes. Common filtering patterns include:

Filter to only EntityChangeEvent_v1 events for notification pipelines
Filter to tag-related changes for tag propagation (category: "TAG")
Filter to specific entity types (entityType: "dataset")
Filter to specific operations (operation: "ADD" or operation: "REMOVE")
Combine multiple criteria for precise targeting

Theoretical Basis

Content-based routing pattern: Events are routed to actions based on their type and content, preventing unnecessary processing. This pattern ensures that action plugins only receive events they can meaningfully act upon, reducing computational overhead and simplifying action implementation. The filter stage sits early in the pipeline (immediately after the source) to discard irrelevant events before any expensive transformation or action processing occurs.

The distinction between ANY semantics for lists and ALL semantics for dictionaries reflects common filtering needs: list matching expresses "match any of these alternatives" (disjunction), while dictionary matching expresses "match all of these criteria" (conjunction). This combination enables powerful filtering expressions without a custom query language.

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment