Implementation:Datahub project Datahub RemovePathPatternUtils

Knowledge Sources	Datahub_project_Datahub
Domains	Spark_Lineage, OpenLineage
Last Updated	2026-02-10 00:00 GMT

Overview

RemovePathPatternUtils is a utility class in the io.openlineage.spark.agent.util package that handles removing path patterns from dataset names in OpenLineage events. It supports two distinct mechanisms:

1. OpenLineage native pattern removal: Uses a regex pattern configured via spark.openlineage.dataset.removePath.pattern with a named group remove to strip dynamic path segments (e.g., partition paths, timestamps) from dataset names.
2. DataHub PathSpec-based removal: Uses DataHub's HdfsPathDataset with the DatahubOpenlineageConfig to apply DataHub-specific path transformations, converting raw HDFS paths into normalized dataset paths according to configured path specs.

The class processes both input and output datasets, creating new dataset objects with cleaned names while preserving all other metadata (namespace, facets).

Source file: metadata-integration/java/acryl-spark-lineage/src/main/java/io/openlineage/spark/agent/util/RemovePathPatternUtils.java (183 lines)

Code Reference

Class Declaration

@Slf4j
public class RemovePathPatternUtils {
  public static final String REMOVE_PATTERN_GROUP = "remove";
  public static final String SPARK_OPENLINEAGE_DATASET_REMOVE_PATH_PATTERN =
      "spark.openlineage.dataset.removePath.pattern";

Key Methods

removeOutputsPathPattern_ol (OpenLineage native)

public static List<OutputDataset> removeOutputsPathPattern_ol(
    OpenLineageContext context, List<OutputDataset> outputs)

Applies the OpenLineage regex-based pattern removal to output datasets. Reads the pattern from Spark configuration key spark.openlineage.dataset.removePath.pattern, compiles it, and removes the named group remove from each dataset name. Only creates a new dataset object if the name actually changes.

removeOutputsPathPattern (DataHub PathSpec)

public static List<OutputDataset> removeOutputsPathPattern(
    OpenLineageContext context, List<OutputDataset> outputs)

Applies DataHub PathSpec-based path transformation to output datasets. Delegates to removePathPattern which loads the DataHub configuration from Spark properties and uses HdfsPathDataset.create for normalization.

removeInputsPathPattern (DataHub PathSpec)

public static List<InputDataset> removeInputsPathPattern(
    OpenLineageContext context, List<InputDataset> inputs)

Same as removeOutputsPathPattern but operates on input datasets. Creates new InputDataset objects with cleaned names while preserving inputFacets.

removePath (private)

private static String removePath(Pattern pattern, String name)

Core regex-based removal logic. Matches the pattern against the dataset name, finds the remove named group, and excises that portion from the string. Returns the original name if no match.

removePathPattern (private)

private static String removePathPattern(String datasetName)

Core DataHub-based removal logic. Loads Spark configuration, constructs a DatahubOpenlineageConfig from spark.datahub.* properties, and passes the dataset name through HdfsPathDataset.create to get the normalized path.

loadSparkConf (private)

private static Optional<SparkConf> loadSparkConf()

Lazily loads and caches the SparkConf from the default Spark session. The caching addresses Spark issue SPARK-29046 where the session may be closed but the configuration is still needed.

I/O Contract

Direction	Type	Description
Input	OpenLineageContext	Context providing access to Spark configuration and the OpenLineage client for building new dataset objects.
Input	List<OutputDataset> or List<InputDataset>	Dataset lists from OpenLineage events whose names may contain dynamic path segments.
Output	List<OutputDataset> or List<InputDataset>	Dataset lists with cleaned names; unchanged datasets are passed through by reference.

Usage Examples

Applying DataHub PathSpec-based removal to output datasets during event processing:

List<OutputDataset> outputs = ...; // from OpenLineage event
List<OutputDataset> cleaned = RemovePathPatternUtils.removeOutputsPathPattern(context, outputs);
// e.g., "hdfs://ns/warehouse/table/year=2024/month=01/day=15"
//     -> "hdfs://ns/warehouse/table" (based on DataHub path spec config)

Using the OpenLineage native regex pattern:

// Spark config: spark.openlineage.dataset.removePath.pattern =
//   "(?<remove>/year=\\d+/month=\\d+/day=\\d+)"
List<OutputDataset> cleaned = RemovePathPatternUtils.removeOutputsPathPattern_ol(context, outputs);
// Strips the /year=.../month=.../day=... suffix from dataset names

Related Pages

• Datahub_project_Datahub_SparkPathUtils - Primary path resolution utility; RemovePathPatternUtils acts as a post-processing step on its output
• Datahub_project_Datahub_RddPathUtils - Extracts raw paths from RDDs before normalization
• Datahub_project_Datahub_SparkConfigParser_ParseSparkConfig - Parses the spark.datahub.* configuration used by the DataHub path removal method
• Datahub_project_Datahub_StreamingDataSourceV2RelationVisitor - May apply path pattern removal to streaming input datasets
• Datahub_project_Datahub_WriteToDataSourceV2Visitor - May apply path pattern removal to streaming output datasets