Implementation:NVIDIA NeMo Curator Resiliparse Extractor

Knowledge Sources	NVIDIA NeMo Curator
Domains	Text Extraction, Boilerplate Removal, HTML Processing, NLP
Last Updated	2026-02-14 00:00 GMT

Overview

ResiliparseExtractor implements HTML text extraction using the Resiliparse library's fast rule-based plain text extractor, enhanced with a stopword density filter for quality control.

Description

The ResiliparseExtractor class extends HTMLExtractorAlgorithm and provides a high-performance alternative to JusText for HTML text extraction. It wraps the Resiliparse HTML2Text module, which is a very fast, rule-based plain text extractor that uses the Resiliparse DOM parser.

The extraction process works as follows:

Text extraction: Calls resiliparse.extract.html2text.extract_plain_text with optional main_content heuristics and alt_texts preservation. This extracts all visible text nodes inside the HTML <body>, skipping <script>, <style>, and other invisible elements.
Paragraph splitting: The extracted text is split by newlines, filtering out empty strings.
Stopword density filtering: For space-separated languages, each paragraph is checked against a stopword density threshold (default 0.32). Only paragraphs meeting the minimum density are retained. This step is skipped for non-spaced languages (Thai, Chinese, Japanese, Korean) where stopword density metrics are unreliable.

NeMo Curator added the stopword density filter on top of the base Resiliparse extraction to improve quality, since research suggests that around 30-40% of typical English text consists of stopwords.

Usage

Use this class when you need fast HTML text extraction, particularly for large-scale Common Crawl processing. It is well-suited for scenarios where processing speed is prioritized, while still maintaining reasonable quality through stopword density filtering.

Code Reference

Source Location

Repository: NeMo-Curator
File: nemo_curator/stages/text/download/html_extractors/resiliparse.py
Lines: 1-79

Signature

class ResiliparseExtractor(HTMLExtractorAlgorithm):
    def __init__(
        self,
        required_stopword_density: float = 0.32,
        main_content: bool = True,
        alt_texts: bool = False,
    ): ...

    def extract_text(
        self,
        html: str,
        stop_words: frozenset[str],
        language: str,
    ) -> list[str] | None: ...

Import

from nemo_curator.stages.text.download.html_extractors.resiliparse import ResiliparseExtractor

I/O Contract

Inputs

Name	Type	Required	Description
required_stopword_density	float	No	Minimum proportion of stopwords required to preserve a paragraph. Defaults to 0.32
main_content	bool	No	Whether to apply heuristics for extracting only "main-content" elements. Defaults to True
alt_texts	bool	No	Whether to preserve alternative text descriptions (e.g., for images). Defaults to False

The extract_text method accepts:

Name	Type	Required	Description
html	str	Yes	Decoded HTML content string
stop_words	frozenset[str]	Yes	Language-specific stop word set for density filtering
language	str	Yes	Detected language name (uppercase, e.g., "ENGLISH")

Outputs

Name	Type	Description
return value	list[str]	List of extracted text paragraphs that meet the stopword density threshold. For non-spaced languages, all non-empty paragraphs are returned

Usage Examples

Basic Usage

from nemo_curator.stages.text.download.html_extractors.resiliparse import ResiliparseExtractor

extractor = ResiliparseExtractor()

html = "<html><body><p>This is the main content of the article.</p></body></html>"
stop_words = frozenset(["the", "is", "of"])
paragraphs = extractor.extract_text(html, stop_words, "ENGLISH")

Custom Stopword Density

from nemo_curator.stages.text.download.html_extractors.resiliparse import ResiliparseExtractor

# Lower threshold to keep more paragraphs
extractor = ResiliparseExtractor(
    required_stopword_density=0.20,
    main_content=True,
    alt_texts=True,
)

Using with CommonCrawlHTMLExtractor

from nemo_curator.stages.text.download.common_crawl.extract import CommonCrawlHTMLExtractor

# Use Resiliparse as the extraction algorithm
extractor = CommonCrawlHTMLExtractor(algorithm="resiliparse")

Related Pages

Environment:NVIDIA_NeMo_Curator_Python_Linux_Base
NVIDIA_NeMo_Curator_CommonCrawl_Extractor - Uses this as a pluggable extraction algorithm
NVIDIA_NeMo_Curator_JusText_Extractor - Default alternative with context-sensitive classification
NVIDIA_NeMo_Curator_Trafilatura_Extractor - Alternative with cascading fallback algorithms

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment