Implementation:Huggingface Datatrove SentenceDedupSignature

Knowledge Sources	Huggingface_Datatrove
Domains	Data Deduplication, NLP
Last Updated	2026-02-14 17:00 GMT

Overview

The sentence deduplication module implements a three-stage pipeline (SentenceDedupSignature, SentenceFindDedups, SentenceDedupFilter) that removes duplicate n-sentence spans from documents, inspired by the C4 dataset methodology, along with SentDedupConfig for configuration and SentenceDedupBuildIndex for cross-dataset deduplication.

Description

Sentence-level deduplication provides a middle ground between whole-document dedup and exact substring dedup. Rather than removing entire documents or arbitrary byte ranges, it identifies and removes repeated spans of consecutive sentences, preserving unique content within documents that share some boilerplate passages.

SentDedupConfig is a dataclass controlling the deduplication behavior: the number of consecutive sentences per span (n_sentences, default 3), whether to split on sentence boundaries or newlines, minimum document word and sentence counts after dedup, minimum words required to actually remove a span, text normalization settings, and hash configuration.

SentenceDedupSignature (Stage 1) splits each document into sentences using a language-aware word tokenizer, computes hashes of sliding windows of n_sentences consecutive sentences (after text normalization), and writes sorted (hash, doc_id, sent_id) signature files partitioned across finder workers by hash range.

SentenceFindDedups (Stage 2) merges all sorted signature files via a heap-based priority queue, identifies matching hashes indicating duplicate sentence spans, and records the (doc_id, sent_id) pairs. It supports optional deduplication against a pre-built index.

SentenceDedupFilter (Stage 3) loads the duplicate sentence indices, removes the flagged sentence spans from each document, and drops documents that fall below minimum word or sentence counts. It optionally writes the original formatted text with markers (>>> and <<<) showing removed spans.

SentenceDedupBuildIndex creates a hash-only index for cross-dataset sentence deduplication.

Usage

Use this module for fine-grained deduplication at the sentence level, particularly when processing web-crawled text that contains repeated boilerplate across documents but also unique content worth preserving.

Code Reference

Source Location

• Repository: Huggingface_Datatrove
• File: src/datatrove/pipeline/dedup/sentence_dedup.py
• Lines: 1-503

Signature

@dataclass
class SentDedupConfig:
    n_sentences: int = 3
    split_sentences: bool = True
    only_dedup_in_index: bool = True
    min_doc_words: int = 50
    min_num_sentences: int = 3
    min_words_to_remove_span: int = 0
    norm_config: TextNormConfig = field(default_factory=TextNormConfig)
    hash_config: HashConfig = field(default_factory=HashConfig)

class SentenceDedupSignature(PipelineStep):
    def __init__(
        self,
        output_folder: DataFolderLike,
        finder_workers: int = 1,
        config: SentDedupConfig = None,
        language: str = Languages.english,
    ):

class SentenceFindDedups(PipelineStep):
    def __init__(
        self,
        data_folder: DataFolderLike,
        output_folder: DataFolderLike,
        index_folder: DataFolderLike = None,
        config: SentDedupConfig = None,
        lines_to_buffer: int = 5,
    ):

class SentenceDedupFilter(PipelineStep):
    def __init__(
        self,
        data_folder: DataFolderLike,
        config: SentDedupConfig = None,
        exclusion_writer: DiskWriter = None,
        language: str = Languages.english,
    ):

Import

from datatrove.pipeline.dedup.sentence_dedup import (
    SentDedupConfig,
    SentenceDedupSignature,
    SentenceFindDedups,
    SentenceDedupFilter,
    SentenceDedupBuildIndex,
)

I/O Contract

Inputs

Name	Type	Required	Description
output_folder	DataFolderLike	Yes (Stage 1)	Folder where sentence hash signatures are saved
finder_workers	int	No	Number of workers for Stage 2 (default: 1)
config	SentDedupConfig	No	Dedup configuration (defaults provided)
language	str	No	Language for sentence tokenization (default: English)
data_folder	DataFolderLike	Yes (Stages 2, 3)	Folder with signature or duplicate files from previous stage
index_folder	DataFolderLike	No	Folder with pre-built index for cross-dataset dedup
exclusion_writer	DiskWriter	No	Writer to save excluded documents with removal markers

Outputs

Name	Type	Description
Signature files	Binary	Sorted (hash, doc_id, sent_id) tuples partitioned by hash range
Duplicate files	Binary	(doc_id, sent_id) pairs flagged as duplicate sentence spans
Filtered documents	DocumentsPipeline	Documents with duplicate sentence spans removed

Usage Examples

Basic Usage

from datatrove.pipeline.dedup.sentence_dedup import (
    SentDedupConfig,
    SentenceDedupSignature,
    SentenceFindDedups,
    SentenceDedupFilter,
)

config = SentDedupConfig(
    n_sentences=3,
    min_doc_words=50,
    min_num_sentences=3,
)

# Stage 1: Generate sentence signatures
stage1 = SentenceDedupSignature(
    output_folder="s3://bucket/dedup/sent_sigs",
    finder_workers=4,
    config=config,
    language="en",
)

# Stage 2: Find duplicate sentence spans
stage2 = SentenceFindDedups(
    data_folder="s3://bucket/dedup/sent_sigs",
    output_folder="s3://bucket/dedup/sent_dups",
    config=config,
)

# Stage 3: Filter duplicate sentences from documents
stage3 = SentenceDedupFilter(
    data_folder="s3://bucket/dedup/sent_dups",
    config=config,
)

Related Pages

• Principle:Huggingface_Datatrove_Sentence_Deduplication