Implementation:NVIDIA NeMo Curator Trafilatura Extractor

Knowledge Sources	NVIDIA NeMo Curator
Domains	Text Extraction, Boilerplate Removal, HTML Processing, NLP
Last Updated	2026-02-14 00:00 GMT

Overview

TrafilaturaExtractor implements HTML text extraction using the Trafilatura library, which combines XPath-based content heuristics with readability-lxml and jusText fallbacks, enhanced with a stopword density filter.

Description

The TrafilaturaExtractor class extends HTMLExtractorAlgorithm and provides high-quality HTML text extraction by leveraging the Trafilatura library. Trafilatura's extraction process follows a cascade of strategies:

1. Content Delimitation: Uses XPath expressions to exclude unwanted HTML elements (e.g., navigation bars) and focus on relevant content (e.g., article body). Extracted HTML nodes are analyzed for relevance based on element type, text length, and link density.
2. Fallback Mechanism: If extraction seems faulty, alternative algorithms (readability-lxml and jusText) are run as backups. These use heuristics like line length, text-to-markup ratio, and HTML depth. Outputs are compared, prioritizing longer extractions with fewer impurities.
3. Baseline Extraction: If all else fails, it searches for text elements that might have been missed, discarding irrelevant content.

The NeMo Curator implementation deep-copies Trafilatura's default configuration and overrides several settings:

• MIN_EXTRACTED_SIZE - Acceptable size in characters to trigger fallbacks (default: 250)
• MIN_EXTRACTED_COMM_SIZE - Minimum size for comment extraction (default: 1)
• MIN_OUTPUT_SIZE - Absolute minimum for main text output (default: 1)
• MIN_OUTPUT_COMM_SIZE - Minimum for comment output (default: 1)
• MAX_TREE_SIZE - Discard documents with too many elements (default: None)
• MIN_DUPLCHECK_SIZE - Minimum size for deduplication (default: 100)
• MAX_REPETITIONS - Maximum allowed duplicate blocks (default: 2)

Deduplication is enabled by default (deduplicate=True). After extraction, a stopword density filter (default threshold 0.32) is applied to keep only paragraphs with sufficient stopword proportion. This filter is skipped for non-spaced languages (Thai, Chinese, Japanese, Korean).

Usage

Use this class when you want maximum content recall from HTML pages, as the cascading fallback mechanism makes it robust against varied HTML structures. It is suitable for scenarios where extraction quality is prioritized over raw speed.

Code Reference

Source Location

• Repository: NeMo-Curator
• File: nemo_curator/stages/text/download/html_extractors/trafilatura.py
• Lines: 1-133

Signature

class TrafilaturaExtractor(HTMLExtractorAlgorithm):
    def __init__(
        self,
        required_stopword_density: float = 0.32,
        min_extracted_size: int = 250,
        min_extracted_comm_size: int = 1,
        min_output_size: int = 1,
        min_output_comm_size: int = 1,
        max_tree_size: int | None = None,
        min_duplcheck_size: int = 100,
        max_repetitions: int = 2,
        **extract_kwargs,
    ): ...

    def extract_text(
        self,
        html: str,
        stop_words: frozenset[str],
        language: str,
    ) -> list[str] | None: ...

Import

from nemo_curator.stages.text.download.html_extractors.trafilatura import TrafilaturaExtractor

I/O Contract

Inputs

Name	Type	Required	Description
required_stopword_density	float	No	Minimum proportion of stopwords required to preserve a paragraph. Defaults to 0.32
min_extracted_size	int	No	Acceptable size in characters used to trigger fallback algorithms. Defaults to 250
min_extracted_comm_size	int	No	Minimum size for comment extraction. Defaults to 1
min_output_size	int	No	Absolute acceptable minimum for main text output. Defaults to 1
min_output_comm_size	int	No	Minimum size for comment output. Defaults to 1
max_tree_size	int or None	No	Maximum number of DOM elements before discarding the document. Defaults to None (no limit)
min_duplcheck_size	int	No	Minimum character count to run deduplication on a block. Defaults to 100
max_repetitions	int	No	Maximum number of allowed duplicate blocks. Defaults to 2
**extract_kwargs	dict	No	Additional keyword arguments passed to trafilatura.extract(). See Trafilatura API docs. Deduplication is set to True by default

The extract_text method accepts:

Name	Type	Required	Description
html	str	Yes	Decoded HTML content string
stop_words	frozenset[str]	Yes	Language-specific stop word set for density filtering
language	str	Yes	Detected language name (uppercase, e.g., "ENGLISH")

Outputs

Name	Type	Description
return value	list[str] or None	List of extracted text paragraphs meeting the stopword density threshold, or None if Trafilatura extraction fails or no qualifying paragraphs are found

Usage Examples

Basic Usage

from nemo_curator.stages.text.download.html_extractors.trafilatura import TrafilaturaExtractor

extractor = TrafilaturaExtractor()

html = "<html><body><article><p>This is the main article content.</p></article></body></html>"
stop_words = frozenset(["the", "is"])
paragraphs = extractor.extract_text(html, stop_words, "ENGLISH")

Custom Configuration

from nemo_curator.stages.text.download.html_extractors.trafilatura import TrafilaturaExtractor

# More lenient extraction with lower thresholds
extractor = TrafilaturaExtractor(
    required_stopword_density=0.20,
    min_extracted_size=100,
    max_tree_size=5000,
    max_repetitions=3,
)

Using with CommonCrawlHTMLExtractor

from nemo_curator.stages.text.download.common_crawl.extract import CommonCrawlHTMLExtractor

# Use Trafilatura as the extraction algorithm with custom kwargs
extractor = CommonCrawlHTMLExtractor(
    algorithm="trafilatura",
    algorithm_kwargs={
        "required_stopword_density": 0.25,
        "min_extracted_size": 150,
    },
)

Passing Additional Trafilatura Arguments

from nemo_curator.stages.text.download.html_extractors.trafilatura import TrafilaturaExtractor

# Pass additional kwargs directly to trafilatura.extract()
extractor = TrafilaturaExtractor(
    include_comments=False,
    include_tables=True,
    favor_recall=True,
)

Related Pages

• Environment:NVIDIA_NeMo_Curator_Python_Linux_Base
• NVIDIA_NeMo_Curator_CommonCrawl_Extractor - Uses this as a pluggable extraction algorithm
• NVIDIA_NeMo_Curator_JusText_Extractor - Default alternative with context-sensitive classification
• NVIDIA_NeMo_Curator_Resiliparse_Extractor - Alternative focused on speed