Implementation:NVIDIA NeMo Curator WARC Iterator

Knowledge Sources	NVIDIA NeMo Curator
Domains	Data Parsing, Web Crawl, Common Crawl, WARC
Last Updated	2026-02-14 00:00 GMT

Overview

CommonCrawlWarcIterator processes downloaded WARC archive files and yields individual web page records containing raw HTML content, URLs, and record identifiers.

Description

The CommonCrawlWarcIterator class extends DocumentIterator and uses the warcio library's ArchiveIterator to stream through WARC (Web ARChive) files. It bridges between the raw downloaded WARC archives and the HTML extraction step by providing structured per-page records.

For each record in the WARC archive, the iterator:

1. Checks that the record type is "response" (skipping other record types such as "request", "metadata", etc.)
2. Reads the full content stream as bytes
3. Extracts the WARC-Record-ID (stripping the surrounding angle brackets and "urn:uuid:" prefix)
4. Extracts the WARC-Target-URI (the original URL of the web page)
5. Yields a dictionary with the extracted fields

The iterator handles corrupted or malformed records gracefully by logging errors and continuing to the next record, ensuring that a single bad record does not halt processing of the entire file.

Usage

Use this class to iterate over downloaded Common Crawl WARC files and produce per-page record dictionaries. It is typically used between the download step (CommonCrawlWARCDownloader) and the extraction step (CommonCrawlHTMLExtractor).

Code Reference

Source Location

• Repository: NeMo-Curator
• File: nemo_curator/stages/text/download/common_crawl/warc_iterator.py
• Lines: 1-57

Signature

class CommonCrawlWarcIterator(DocumentIterator):
    def iterate(self, file_path: str) -> Iterator[dict[str, Any]]: ...

    def output_columns(self) -> list[str]: ...

Import

from nemo_curator.stages.text.download.common_crawl.warc_iterator import CommonCrawlWarcIterator

I/O Contract

Inputs

Name	Type	Required	Description
file_path	str or Path	Yes	Path to a downloaded WARC file (typically a compressed .warc.gz file)

Outputs

Each yielded record is a dictionary with the following fields:

Name	Type	Description
url	str	The WARC-Target-URI header value, i.e., the original URL of the web page
warc_id	str	The WARC-Record-ID with "urn:uuid:" prefix and angle brackets stripped
source_id	str	The filename of the WARC file being processed
content	bytes	The raw HTTP response body (typically HTML content)

Usage Examples

Basic Iteration

from nemo_curator.stages.text.download.common_crawl.warc_iterator import CommonCrawlWarcIterator

iterator = CommonCrawlWarcIterator()

for record in iterator.iterate("/data/common_crawl/raw/CC-MAIN-2024-01.warc.gz"):
    print(f"URL: {record['url']}")
    print(f"WARC ID: {record['warc_id']}")
    print(f"Content size: {len(record['content'])} bytes")

Combined with Extractor

from nemo_curator.stages.text.download.common_crawl.warc_iterator import CommonCrawlWarcIterator
from nemo_curator.stages.text.download.common_crawl.extract import CommonCrawlHTMLExtractor

iterator = CommonCrawlWarcIterator()
extractor = CommonCrawlHTMLExtractor(algorithm="justext")

for record in iterator.iterate("/data/common_crawl/raw/CC-MAIN-2024-01.warc.gz"):
    result = extractor.extract(record)
    if result is not None:
        print(f"Language: {result['language']}, Text length: {len(result['text'])}")

Related Pages

• Environment:NVIDIA_NeMo_Curator_Python_Linux_Base
• NVIDIA_NeMo_Curator_CommonCrawl_Downloader - Downloads the WARC files that this iterator processes
• NVIDIA_NeMo_Curator_CommonCrawl_Extractor - Extracts text from the records yielded by this iterator
• NVIDIA_NeMo_Curator_CommonCrawlDownloadExtractStage - Orchestrates the full Common Crawl pipeline