Implementation:NVIDIA NeMo Curator Wikipedia URLGenerator

Knowledge Sources	NVIDIA NeMo Curator
Domains	URL Generation, Wikipedia, Data Discovery
Last Updated	2026-02-14 00:00 GMT

Overview

WikipediaUrlGenerator generates URLs for Wikipedia dump files from the Wikimedia dumps server, supporting automatic discovery of the latest available dump or use of a specific dump date.

Description

The WikipediaUrlGenerator class extends URLGenerator as a dataclass and provides URL discovery for the Wikipedia download pipeline. It interacts with the Wikimedia Foundation's dump infrastructure at dumps.wikimedia.org.

The URL generation process supports two modes:

Automatic latest dump discovery (dump_date=None):

Fetches the index page for the specified language wiki (e.g., https://dumps.wikimedia.org/enwiki)
Parses the HTML using BeautifulSoup to find all available dump dates
Iterates through dumps in reverse chronological order
For each candidate, fetches dumpstatus.json and checks if the articlesmultistreamdump job has status "done"
Selects the most recent completed dump

Specific dump date (dump_date provided):

Directly fetches dumpstatus.json for the specified date
Validates that the dump exists and the articlesmultistreamdump job is complete
Raises ValueError if the dump is not found or not finished

Once a valid dump is selected, the generator extracts all XML file URLs from the articlesmultistreamdump job's file list, filtering to only include files with "xml" in their name.

All HTTP requests use a 30-second timeout (REQUEST_TIMEOUT constant).

Usage

Use this class to discover and generate download URLs for Wikipedia dump files. It is typically the first step in the Wikipedia download pipeline, producing URLs that are then passed to WikipediaDownloader.

Code Reference

Source Location

Repository: NeMo-Curator
File: nemo_curator/stages/text/download/wikipedia/url_generation.py
Lines: 1-114

Signature

@dataclass
class WikipediaUrlGenerator(URLGenerator):
    language: str = "en"
    dump_date: str | None = None
    wikidumps_index_prefix: str = "https://dumps.wikimedia.org"

    def generate_urls(self) -> list[str]: ...

Import

from nemo_curator.stages.text.download.wikipedia.url_generation import WikipediaUrlGenerator

I/O Contract

Inputs

Name	Type	Required	Description
language	str	No	Language code for the Wikipedia dump (e.g., "en", "de", "fr", "ja"). Defaults to "en"
dump_date	str or None	No	Specific dump date in YYYYMMDD format (e.g., "20240101"). If None (default), automatically discovers the latest available dump
wikidumps_index_prefix	str	No	Base URL for the Wikimedia dumps server. Defaults to "https://dumps.wikimedia.org"

Outputs

Name	Type	Description
return value	list[str]	From `generate_urls()`: a list of fully-qualified URLs pointing to Wikipedia multistream XML dump files (.bz2)

Usage Examples

Auto-discover Latest Dump

from nemo_curator.stages.text.download.wikipedia.url_generation import WikipediaUrlGenerator

# Automatically find the latest completed English Wikipedia dump
generator = WikipediaUrlGenerator(language="en")
urls = generator.generate_urls()
print(f"Found {len(urls)} dump files")
for url in urls[:3]:
    print(url)

Specific Dump Date

from nemo_curator.stages.text.download.wikipedia.url_generation import WikipediaUrlGenerator

# Use a specific dump date
generator = WikipediaUrlGenerator(
    language="en",
    dump_date="20240101",
)
urls = generator.generate_urls()

Non-English Wikipedia

from nemo_curator.stages.text.download.wikipedia.url_generation import WikipediaUrlGenerator

# Generate URLs for German Wikipedia
generator = WikipediaUrlGenerator(language="de")
urls = generator.generate_urls()
print(f"Found {len(urls)} German Wikipedia dump files")

Full Download Pipeline

from nemo_curator.stages.text.download.wikipedia.url_generation import WikipediaUrlGenerator
from nemo_curator.stages.text.download.wikipedia.download import WikipediaDownloader

# Step 1: Discover URLs
url_gen = WikipediaUrlGenerator(language="en")
urls = url_gen.generate_urls()

# Step 2: Download files
downloader = WikipediaDownloader(download_dir="/data/wikipedia/raw", verbose=True)
# The downloader is used within the WikipediaDownloadExtractStage pipeline

Related Pages

Environment:NVIDIA_NeMo_Curator_Python_Linux_Base
NVIDIA_NeMo_Curator_Wikipedia_Downloader - Downloads the dump files discovered by this generator
NVIDIA_NeMo_Curator_Wikipedia_Iterator - Iterates over downloaded dump files
NVIDIA_NeMo_Curator_Wikipedia_Extractor - Extracts text from Wikipedia articles

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment