Principle:Huggingface Datatrove Sentence Deduplication
| Knowledge Sources | |
|---|---|
| Domains | Data Deduplication, NLP |
| Last Updated | 2026-02-14 17:00 GMT |
Overview
Sentence Deduplication is the principle of removing duplicate spans of consecutive sentences from documents, preserving unique content within documents that share repeated boilerplate passages.
Description
Originally described in the C4 dataset paper ("Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer", Raffel et al., JMLR 2020), sentence deduplication discards all but one occurrence of any three-sentence span appearing more than once in the dataset. This provides a fine-grained deduplication method that sits between whole-document dedup (which drops entire documents) and exact substring dedup (which operates at the byte level).
The key insight is that many web documents share boilerplate text (navigation menus, footer text, legal disclaimers) but also contain unique valuable content. By operating at the sentence level, the method can strip away repeated passages while preserving the unique portions of each document. This maximizes data retention while still removing the redundancy that can bias language model training.
Usage
Apply sentence deduplication when processing web-crawled datasets where documents frequently share boilerplate passages but contain unique content worth preserving. It is particularly effective as a complement to whole-document deduplication.
Theoretical Basis
The sentence deduplication algorithm uses a sliding window hash approach across three stages:
- N-gram Sentence Hashing: Documents are split into sentences (using language-aware tokenizers or newline splitting). A sliding window of
nconsecutive sentences (default 3) moves across each document. Each window is normalized (lowercased, whitespace-stripped) and hashed to produce a fixed-size signature. The (hash, doc_id, sent_id) triples are sorted and partitioned by hash range.
- Heap-Based Duplicate Detection: All sorted signature files are merged using a k-way merge with a min-heap. Consecutive entries with matching hashes indicate duplicate sentence spans. The first occurrence is retained; subsequent occurrences are recorded as duplicates with their (doc_id, sent_id) pairs.
- Span Removal with Thresholds: When removing flagged sentence spans, the algorithm tracks which sentence indices to drop and extends the removal window by
n_sentences - 1positions (since a 3-sentence match at positionimeans sentencesithroughi+2are duplicated). After removal, documents are checked against minimum thresholds:- Minimum document words: Documents with too few words remaining are discarded
- Minimum number of sentences: Documents with too few sentences remaining are discarded
- Minimum words to remove span: Short matching spans (below this threshold) are kept to avoid fragmenting text
- Cross-Dataset Deduplication: A pre-built index of sentence hashes from a reference dataset allows deduplication of new data against previously processed corpora without reprocessing.