Implementation:NVIDIA NeMo Curator Wikipedia Extractor

Knowledge Sources	NVIDIA NeMo Curator
Domains	NLP, Text Extraction, Wikipedia, Data Curation
Last Updated	2026-02-14 00:00 GMT

Overview

Extracts and cleans plain text from Wikipedia MediaWiki markup, handling multilingual namespace prefixes for media and category links across approximately 200 languages.

Description

The WikipediaExtractor class extends DocumentExtractor and provides the core text extraction logic for converting raw Wikipedia wikitext markup into clean, readable plain text. The implementation is based on the HuggingFace Wikipedia dataset preparation code.

The module contains two large dictionaries that map language codes to localized namespace prefixes:

• MEDIA_ALIASES: Maps approximately 200 language codes to their localized file/image/media namespace prefixes (e.g., "Datei" and "Bild" for German, "Fichier" for French, "ファイル" and "画像" for Japanese)
• CAT_ALIASES: Maps approximately 200 language codes to their localized category namespace prefixes (e.g., "Kategorie" for German, "Catégorie" for French, "カテゴリ" for Japanese)

The extraction process works as follows:

1. Regex Pattern Creation (_create_filters): Builds three compiled regex patterns -- one for removing magic words (e.g., ), one for identifying file/image/media links using language-specific prefixes, and one for cleaning category links by stripping their prefixes.
2. Filter Function Creation (_create_filter_functions): Generates closure-based filter functions (rm_wikilink, rm_tag, is_category, try_replace_obj, try_remove_obj) that safely handle object removal with contextlib.suppress(ValueError).
3. Section Processing (_process_sections): Iterates over all sections of the parsed wikicode, removes media/image wikilinks, cleans category link prefixes, strips ref and table tags, removes magic words, and joins cleaned section text with double newlines.
4. Extraction (extract): Parses raw wikitext using mwparserfromhell, applies all filters, and returns a dictionary with cleaned text and metadata fields.

Usage

Use this extractor when building a text curation pipeline that ingests Wikipedia XML dump data. The extractor expects records with raw_content containing MediaWiki markup and produces clean text output suitable for downstream NLP tasks. Initialize with the appropriate language code to ensure correct handling of localized namespace prefixes.

Code Reference

Source Location

• Repository: NeMo-Curator
• File: nemo_curator/stages/text/download/wikipedia/extract.py
• Lines: 1-716

Signature

class WikipediaExtractor(DocumentExtractor):
    def __init__(self, language: str = "en"): ...
    def _create_filters(self) -> tuple[re.Pattern[str], re.Pattern[str], re.Pattern[str]]: ...
    def _create_filter_functions(self, re_rm_wikilink, re_clean_wikilink) -> tuple: ...
    def _process_sections(self, wikicode, re_rm_magic, rm_wikilink, rm_tag, is_category, try_replace_obj, try_remove_obj) -> str: ...
    def extract(self, record: dict[str, Any]) -> dict[str, Any] | None: ...
    def input_columns(self) -> list[str]: ...
    def output_columns(self) -> list[str]: ...

Import

from nemo_curator.stages.text.download.wikipedia.extract import WikipediaExtractor

I/O Contract

Inputs

Name	Type	Required	Description
language	str	No (default: "en")	Language code for the Wikipedia articles, used to select the correct media and category namespace aliases.
record	dict[str, Any]	Yes	A dictionary containing fields: title, id, url, language, source_id, and raw_content (the MediaWiki markup text).

Outputs

Name	Type	Description
result	None	A dictionary with keys text (cleaned plain text), title, id, url, language, and source_id. Returns None if extraction fails or raw_content is empty.

Usage Examples

Basic Usage

from nemo_curator.stages.text.download.wikipedia.extract import WikipediaExtractor

# Create an extractor for English Wikipedia
extractor = WikipediaExtractor(language="en")

# Process a record from a Wikipedia dump
record = {
    "title": "Python (programming language)",
    "id": "12345",
    "url": "https://en.wikipedia.org/wiki/Python_(programming_language)",
    "language": "en",
    "source_id": "enwiki-20240101",
    "raw_content": "'''Python''' is a [[programming language]]...",
}

result = extractor.extract(record)
if result is not None:
    print(result["text"])  # Clean plain text

Multilingual Usage

from nemo_curator.stages.text.download.wikipedia.extract import WikipediaExtractor

# Create an extractor for German Wikipedia
de_extractor = WikipediaExtractor(language="de")

# The extractor will correctly handle German namespace prefixes
# such as "Datei:" for files and "Kategorie:" for categories
record = {
    "title": "Berlin",
    "id": "67890",
    "url": "https://de.wikipedia.org/wiki/Berlin",
    "language": "de",
    "source_id": "dewiki-20240101",
    "raw_content": "'''Berlin''' ist die [[Hauptstadt]]...",
}

result = de_extractor.extract(record)

Column Definitions

from nemo_curator.stages.text.download.wikipedia.extract import WikipediaExtractor

extractor = WikipediaExtractor(language="en")

# Check expected input and output columns
print(extractor.input_columns())
# ['title', 'id', 'url', 'language', 'source_id', 'raw_content']

print(extractor.output_columns())
# ['text', 'title', 'id', 'url', 'language', 'source_id']

Related Pages

• Environment:NVIDIA_NeMo_Curator_Python_Linux_Base