Implementation:NVIDIA NeMo Curator CommonCrawl Downloader

Knowledge Sources	NVIDIA NeMo Curator
Domains	Data Acquisition, Web Crawl, Common Crawl
Last Updated	2026-02-14 00:00 GMT

Overview

CommonCrawlWARCDownloader downloads WARC (Web ARChive) files from the Common Crawl dataset to a local directory, supporting both HTTP (wget) and S3 (s5cmd) download methods.

Description

The CommonCrawlWARCDownloader class extends DocumentDownloader and serves as the core data acquisition component for the Common Crawl pipeline. It handles the actual file transfer of compressed WARC archives from Common Crawl's public infrastructure.

The downloader supports two transport mechanisms:

• wget (default): Downloads files over HTTP from data.commoncrawl.org.
• s5cmd (AWS S3): Downloads from the s3://commoncrawl/ bucket using the s5cmd tool, which can be significantly faster for bulk downloads.

When using the S3 transport, the constructor validates that s5cmd is installed on the system and raises a RuntimeError if it is not found. Output filenames are derived from the URL path by replacing slashes with hyphens to create a flat file structure in the download directory.

Each download operation is executed as a subprocess call, returning a (success, error_message) tuple. The verbose parameter controls whether subprocess output is logged or suppressed.

Usage

Use this class when you need to download raw WARC files from Common Crawl as part of a data curation pipeline. It is typically used in conjunction with CommonCrawlWarcIterator (to parse the WARC files) and CommonCrawlHTMLExtractor (to extract text from the HTML content).

Code Reference

Source Location

• Repository: NeMo-Curator
• File: nemo_curator/stages/text/download/common_crawl/download.py
• Lines: 1-91

Signature

class CommonCrawlWARCDownloader(DocumentDownloader):
    def __init__(
        self,
        download_dir: str,
        use_aws_to_download: bool = False,
        verbose: bool = False,
    ): ...

    def _get_output_filename(self, url: str) -> str: ...

    def _download_to_path(self, url: str, path: str) -> tuple[bool, str | None]: ...

Import

from nemo_curator.stages.text.download.common_crawl.download import CommonCrawlWARCDownloader

I/O Contract

Inputs

Name	Type	Required	Description
download_dir	str	Yes	Path to the local directory where raw compressed WARC files will be stored
use_aws_to_download	bool	No	If True, uses s5cmd to download from S3. If False (default), uses wget over HTTP
verbose	bool	No	If True, logs stdout and stderr of the download command (s5cmd or wget)

Outputs

Name	Type	Description
return value	tuple[bool, str or None]	From _download_to_path: a tuple of (success, error_message). If success is True, error_message is None. If False, error_message contains the error details
downloaded file	WARC file	Compressed WARC archive file saved to the download directory

Usage Examples

Basic Usage with wget

from nemo_curator.stages.text.download.common_crawl.download import CommonCrawlWARCDownloader

downloader = CommonCrawlWARCDownloader(
    download_dir="/data/common_crawl/raw",
    verbose=True,
)

Using S3 Download

from nemo_curator.stages.text.download.common_crawl.download import CommonCrawlWARCDownloader

# Requires s5cmd to be installed on the system
downloader = CommonCrawlWARCDownloader(
    download_dir="/data/common_crawl/raw",
    use_aws_to_download=True,
    verbose=False,
)

Related Pages

• Environment:NVIDIA_NeMo_Curator_Python_Linux_Base
• NVIDIA_NeMo_Curator_CommonCrawl_Extractor - Extracts text from downloaded WARC files
• NVIDIA_NeMo_Curator_WARC_Iterator - Iterates over downloaded WARC files
• NVIDIA_NeMo_Curator_CommonCrawl_URLGenerator - Generates URLs for WARC files to download
• NVIDIA_NeMo_Curator_CommonCrawlDownloadExtractStage - Orchestrates the full Common Crawl pipeline