Implementation:Datahub project Datahub Generate Docs Dir

Knowledge Sources	Datahub_project_Datahub
Domains	Documentation, BuildTooling, ContentGeneration
Last Updated	2026-02-10 00:00 GMT

Overview

TypeScript build script that discovers, preprocesses, and transforms all Markdown documentation files from across the DataHub monorepo into a unified output directory suitable for Docusaurus consumption.

Description

The generateDocsDir.ts script is the core documentation build pipeline for DataHub. It must be executed from within the docs-website directory and performs a multi-step transformation on every Markdown file in the repository. The script first discovers all Markdown files via git ls-files, including generated documentation from docs/generated/, and optionally includes untracked files when not running in CI. It applies a set of filter patterns to exclude irrelevant directories such as .github/, docs-website/, contrib/, datahub-kubernetes/, archived ingestion docs, and test files.

For each discovered Markdown file, the script applies a series of transformations using the gray-matter library for frontmatter manipulation: Title guessing extracts the first H1 header as the document title (with hardcoded overrides for specific files like README.md), and strips "DataHub" or "About DataHub" prefixes from sidebar labels. Slug generation creates URL-friendly paths, removing docs/ prefixes and /README suffixes. URL rewriting transforms absolute GitHub links, local file references, and image paths into the correct format for the hosted documentation site, converting references to non-Markdown files (Java, Python, YAML, etc.) into GitHub browse URLs and rebasing image paths for Docusaurus static asset bundling. Inline directives allow embedding external file contents into Markdown via the Template:Inline /path/to/file syntax. Command output directives execute shell commands and embed their output via Template:Command-output .... Special comment blocks between <!--HOSTED_DOCS_ONLY and HOSTED_DOCS_ONLY--> are enabled by stripping the comment markers.

The script also generates a dynamic releases page by fetching release data from the GitHub API via Octokit (with rate limiting and retry support), constructing a summary table and detailed release notes for releases within the last three months. Finally, it validates that all processed Markdown files are accounted for in sidebars.js, warning about any files missing from the sidebar configuration. Auto-generated sidebar directories (such as docs/generated/metamodel and docs/generated/ingestion) are exempt from this check.

Usage

This script is run as part of the documentation build process, typically before Docusaurus builds the site. It populates the docs/ output directory (which maps to the genDocs path referenced in docusaurus.config.js) with processed Markdown files ready for rendering. It should be executed whenever documentation content changes need to be reflected in the hosted documentation site.

Code Reference

Source Location

• Repository: Datahub_project_Datahub
• File: docs-website/generateDocsDir.ts

Signature

// Key functions exported/used internally:
function list_markdown_files(): string[]
function get_id(filepath: string): string
function get_slug(filepath: string): string
function actually_in_sidebar(filepath: string): boolean
function accounted_for_in_sidebar(filepath: string): boolean
function markdown_guess_title(contents: matter.GrayMatterFile<string>, filepath: string): void
function markdown_add_edit_url(contents: matter.GrayMatterFile<string>, filepath: string): void
function markdown_add_slug(contents: matter.GrayMatterFile<string>, filepath: string): void
function markdown_rewrite_urls(contents: matter.GrayMatterFile<string>, filepath: string): void
function markdown_enable_specials(contents: matter.GrayMatterFile<string>, filepath: string): void
function markdown_process_inline_directives(contents: matter.GrayMatterFile<string>, filepath: string): void
function markdown_process_command_output(contents: matter.GrayMatterFile<string>, filepath: string): void
async function generate_releases_markdown(): Promise<matter.GrayMatterFile<string>>
function write_markdown_file(contents: matter.GrayMatterFile<string>, output_filepath: string): void

Import

// Dependencies
import { execSync } from "child_process";
import matter from "gray-matter";
import * as fs from "fs";
import * as path from "path";
import { Octokit } from "@octokit/rest";
import { throttling } from "@octokit/plugin-throttling";
import { retry } from "@octokit/plugin-retry";

// Execute from docs-website directory:
// npx ts-node generateDocsDir.ts

I/O Contract

Inputs

Name	Type	Required	Description
Repository Markdown files	.md files	Yes	All tracked and generated Markdown files discovered via git ls-files
sidebars.js	JavaScript module	Yes	Sidebar definition used for validation that all docs are accounted for
CI environment variable	Environment variable	No	When set, excludes untracked files from processing
GitHub API	REST API	No	Used to fetch release data for the releases page via Octokit

Outputs

Name	Type	Description
docs/ directory	Directory of Markdown files	Processed Markdown files with updated frontmatter, rewritten URLs, and injected content
docs/releases.md	Markdown file	Auto-generated releases page with summary table and recent release notes
Console warnings	Text	Warnings for files not accounted for in the sidebar configuration

Usage Examples

# Run the documentation generation script
cd docs-website
npx ts-node generateDocsDir.ts

# The script produces output in docs-website/docs/ which is
# referenced as genDocs in docusaurus.config.js

<!-- Inline directive example in a Markdown source file -->
```yaml
{{ inline /metadata-ingestion/examples/recipe.yml show_path_as_comment }}
```

<!-- Command output directive example -->
{{ command-output datahub --help }}

<!-- Hosted docs only content -->
<!--HOSTED_DOCS_ONLY
This content only appears on the hosted documentation site.
HOSTED_DOCS_ONLY-->