Implementation:Datajuicer Data juicer Build Op Doc
| Knowledge Sources | |
|---|---|
| Domains | Documentation, Code Generation, Pre-commit Hooks |
| Last Updated | 2026-02-14 16:00 GMT |
Overview
Pre-commit hook script that auto-generates the Operators.md documentation page by scanning operator source code, extracting class definitions, tags, and docstrings via AST parsing, and producing bilingual (English/Chinese) markdown tables.
Description
This module automates operator documentation maintenance so the Operators.md reference stays synchronized with actual operator implementations. It scans all operator Python files using Python's AST module to extract class names, docstrings, and tags (modality, resource, model). It categorizes operators by type (formatter, mapper, filter, deduplicator, selector, grouper, aggregator, pipeline) and generates markdown tables with tag icons, descriptions, and links to detailed info pages.
Key classes:
- OPRecord -- Data class representing an operator record with type, name, description, tags, test path, info link, and reference link.
- ClassVisitor -- AST node visitor that extracts class names and their first-sentence docstrings from Python source files.
The script also supports bilingual documentation by batch-translating English descriptions to Chinese using the Alibaba translator via the `translators` library. It detects usability tags (alpha/beta/stable) based on the existence of unit tests and preserves manually-set stable tags from previous documentation versions.
Usage
This script is intended to be run as a pre-commit hook. When operator source files change, it regenerates docs/Operators.md. If the documentation changes, the hook exits with a non-zero status to signal that the file was updated and needs to be committed.
Code Reference
Source Location
- Repository: Datajuicer_Data_juicer
- File: .pre-commit-hooks/build_op_doc.py
- Lines: 1-656
Signature
class OPRecord:
def __init__(self, type: str, name: str, desc: str, desc_zh: str = None,
tags: List[str] = None, test: str = None,
info: str = None, ref: str = None):
class ClassVisitor(ast.NodeVisitor):
def __init__(self):
def visit_ClassDef(self, node: ast.ClassDef) -> Any:
def get_class_docs(self):
def main():
Import
import ast
import json
import os
import re
from pathlib import Path
from typing import Any, List
import translators as ts
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| OP source files | Python files | Yes | Operator Python source files under data_juicer/ops/ and data_juicer/format/ |
| tag_mappings.json | JSON file | Yes | Mapping of tag names to icons and descriptions |
| Existing Operators.md | Markdown file | No | Previous documentation for preserving Chinese translations and stable tags |
Outputs
| Name | Type | Description |
|---|---|---|
| docs/Operators.md | Markdown file | Generated bilingual operator documentation with tables, tag icons, and links |
| exit code | int | 0 if no changes, 1 if documentation was updated |
Usage Examples
# As a pre-commit hook (configured in .pre-commit-config.yaml):
# The script is run automatically before each commit.
# Manual execution:
python .pre-commit-hooks/build_op_doc.py
# Programmatic usage of key functions:
from build_op_doc import get_class_and_docstring, analyze_tag_from_code
# Extract class names and docstrings from an operator file
docs = get_class_and_docstring("data_juicer/ops/filter/text_length_filter.py")
# Returns: [("TextLengthFilter", "Filter to keep samples with text length within a specific range.")]
# Analyze tags from operator code
tags = analyze_tag_from_code("data_juicer/ops/filter/text_length_filter.py")
# Returns: ["text", "cpu"]