Implementation:NVIDIA NeMo Curator Wikipedia Downloader

Knowledge Sources	NVIDIA NeMo Curator
Domains	Data Acquisition, Wikipedia, Download
Last Updated	2026-02-14 00:00 GMT

Overview

WikipediaDownloader downloads Wikipedia dump files (.bz2 compressed XML archives) from wikimedia.org to a local directory using wget.

Description

The WikipediaDownloader class extends DocumentDownloader and serves as the core data acquisition component for the Wikipedia pipeline. It handles the file transfer of bz2-compressed XML dump archives from the Wikimedia Foundation's dump servers.

Key characteristics:

• Download method: Uses wget via subprocess to download files. Unlike the Common Crawl downloader, it does not support S3 downloads since Wikipedia dumps are only available from Wikimedia servers.
• Filename generation: Output filenames are derived from the URL path by stripping the leading slash and replacing path separators with hyphens to create a flat file structure.
• Rate limiting: The num_workers_per_node method returns 2, limiting concurrent downloads to be respectful of Wikimedia's servers and avoid overloading their infrastructure.
• Error handling: Each download operation returns a (success, error_message) tuple. stderr is captured when verbose mode is off to provide meaningful error messages on failure.

Usage

Use this class when you need to download raw Wikipedia dump files as part of a data curation pipeline. It is typically used in conjunction with WikipediaUrlGenerator (to discover dump URLs) and WikipediaIterator (to parse the downloaded dump files).

Code Reference

Source Location

• Repository: NeMo-Curator
• File: nemo_curator/stages/text/download/wikipedia/download.py
• Lines: 1-77

Signature

class WikipediaDownloader(DocumentDownloader):
    def __init__(
        self,
        download_dir: str,
        verbose: bool = False,
    ): ...

    def _get_output_filename(self, url: str) -> str: ...

    def _download_to_path(self, url: str, path: str) -> tuple[bool, str | None]: ...

    def num_workers_per_node(self) -> int | None: ...

Import

from nemo_curator.stages.text.download.wikipedia.download import WikipediaDownloader

I/O Contract

Inputs

Name	Type	Required	Description
download_dir	str	Yes	Path to the local directory where raw compressed .bz2 files will be stored
verbose	bool	No	If True, logs stdout and stderr of the wget command. Defaults to False

Outputs

Name	Type	Description
return value	tuple[bool, str or None]	From _download_to_path: a tuple of (success, error_message). If success is True, error_message is None
downloaded file	.bz2 file	Compressed Wikipedia XML dump file saved to the download directory
num_workers_per_node	int	Returns 2, limiting concurrent downloads per node to be respectful of Wikimedia servers

Usage Examples

Basic Usage

from nemo_curator.stages.text.download.wikipedia.download import WikipediaDownloader

downloader = WikipediaDownloader(
    download_dir="/data/wikipedia/raw",
    verbose=True,
)

In a Pipeline

from nemo_curator.stages.text.download.wikipedia.download import WikipediaDownloader
from nemo_curator.stages.text.download.wikipedia.url_generation import WikipediaUrlGenerator

# First generate URLs
url_gen = WikipediaUrlGenerator(language="en")
urls = url_gen.generate_urls()

# Then download the dump files
downloader = WikipediaDownloader(download_dir="/data/wikipedia/raw")

Related Pages

• Environment:NVIDIA_NeMo_Curator_Python_Linux_Base
• NVIDIA_NeMo_Curator_Wikipedia_URLGenerator - Generates the URLs for dump files to download
• NVIDIA_NeMo_Curator_Wikipedia_Iterator - Iterates over the downloaded dump files
• NVIDIA_NeMo_Curator_Wikipedia_Extractor - Extracts text from Wikipedia articles