Implementation:Datahub project Datahub InstitutionalMemoryVisitor

Knowledge Sources	Datahub_project_Datahub
Domains	Protobuf_Integration, Institutional_Knowledge
Last Updated	2026-02-10 00:00 GMT

Overview

Description

InstitutionalMemoryVisitor is a visitor that extracts institutional memory metadata (links, references) from protobuf schema comments. It implements ProtobufModelVisitor<InstitutionalMemoryMetadata> and processes both graph-level (message-level) and field-level comments.

The visitor extracts three categories of links from protobuf comments:

GitHub team references -- Matches @organization/team-name patterns and generates links to https://github.com/orgs/{org}/teams/{team}.
Slack channel references -- Matches #channel-name patterns and generates Slack redirect URLs using the configured team ID.
Inline URLs -- Matches any http, https, ftp, or file URLs found in the comment text.

For field-level comments, only inline URLs are extracted, and only for fields belonging to the root message.

The class includes an internal MatcherStream utility that adapts Java's Matcher into a Stream<MatchResult> via a custom Spliterator.

Usage

Configured as one of the institutionalMemoryMetadataVisitors in DatasetVisitor. Requires optional Slack team ID and GitHub organization name for link generation.

Code Reference

Source Location

metadata-integration/java/datahub-protobuf/src/main/java/datahub/protobuf/visitors/dataset/InstitutionalMemoryVisitor.java

Signature

public class InstitutionalMemoryVisitor
    implements ProtobufModelVisitor<InstitutionalMemoryMetadata> {

    public static final String TEAM_DESC = "Github Team";
    public static final String SLACK_CHAN_DESC = "Slack Channel";

    public InstitutionalMemoryVisitor(
        @Nullable String slackTeamId, @Nullable String githubOrganization)

    @Override
    public Stream<InstitutionalMemoryMetadata> visitGraph(VisitContext context)

    @Override
    public Stream<InstitutionalMemoryMetadata> visitField(ProtobufField field, VisitContext context)
}

Import

import datahub.protobuf.visitors.dataset.InstitutionalMemoryVisitor;

I/O Contract

Inputs

Parameter	Type	Description
`slackTeamId`	`String` (nullable)	Slack workspace team ID for generating Slack redirect URLs
`githubOrganization`	`String` (nullable)	GitHub organization name for matching `@org/team` references

visitGraph inputs:

VisitContext -- provides the root message and audit stamp

visitField inputs:

ProtobufField -- the field whose comments to scan
VisitContext -- provides the graph and audit stamp

Outputs

Method	Return Type	Description
`visitGraph`	`Stream<InstitutionalMemoryMetadata>`	GitHub team link, Slack channel link, and inline URLs from the root message comment
`visitField`	`Stream<InstitutionalMemoryMetadata>`	Inline URLs from the field comment (only for fields of the root message)

Each InstitutionalMemoryMetadata entry contains:

url -- the generated or extracted URL
description -- contextual label (e.g., "Github Team", "Slack Channel", or "{MessageName} Reference {N}")
createStamp -- the audit stamp from the visit context

Usage Examples

InstitutionalMemoryVisitor visitor =
    new InstitutionalMemoryVisitor("T024F4EL1", "my-github-org");

// Used as a sub-visitor in DatasetVisitor
DatasetVisitor datasetVisitor = DatasetVisitor.builder()
    .institutionalMemoryMetadataVisitors(List.of(visitor))
    .build();

Related Pages

Datahub_project_Datahub_ProtobufDatasetVisitor -- Parent visitor that orchestrates this sub-visitor
Datahub_project_Datahub_ProtobufMessage -- Root message whose comments are scanned

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment