Implementation:Huggingface Datatrove SingleBloomFilter

Knowledge Sources	Huggingface_Datatrove
Domains	Deduplication, Data Processing
Last Updated	2026-02-14 17:00 GMT

Overview

Implements single-pass Bloom filter-based deduplication that identifies and removes near-duplicate documents using n-gram shingle hashing with configurable false positive rates.

Description

SingleBloomFilter is a streaming deduplication pipeline step that uses a Bloom filter to detect near-duplicate documents in a single pass over the data. It extends PipelineStep and processes documents one at a time, maintaining a fixed-size bit vector that grows no larger regardless of dataset size.

For each document, the step tokenizes and normalizes the text using simplify_text, generates n-gram shingles (default 13-grams), and hashes each shingle to produce a 32-bit hash value. Each hash is then mapped to k positions in the bit vector using k universal hash functions, parametrized with random coefficients mod a Mersenne prime (2^61 - 1). If all k bits for a shingle are already set (the shingle was "seen before"), it counts as a duplicate shingle. If the fraction of duplicate shingles exceeds the duplicate_threshold (default 0.8), the entire document is classified as a duplicate and dropped.

The BloomFilterConfig dataclass configures the filter size (m_bytes), number of hash functions (k, auto-computed if not provided based on expected number of elements), n-gram size, duplicate threshold, and hash/normalization settings. Helper functions get_optimal_k and get_false_positive_prob compute the optimal number of hash functions and the expected false positive probability given the filter parameters.

The step optionally saves excluded documents via an exclusion_writer and can persist the Bloom filter state to disk for later reuse.

Usage

Use SingleBloomFilter for memory-efficient, single-pass deduplication when processing large datasets on a single node. It is well-suited for scenarios where exact deduplication is too expensive and approximate near-duplicate detection is acceptable.

Code Reference

Source Location

• Repository: Huggingface_Datatrove
• File: src/datatrove/pipeline/dedup/bloom_filter.py
• Lines: 1-210

Signature

@dataclass
class BloomFilterConfig:
    m_bytes: int
    k: int = None
    expected_elements: int = None
    duplicate_threshold: float = 0.8
    n_grams: int = 13
    seed: int = 0
    norm_config: TextNormConfig = field(default_factory=TextNormConfig)
    hash_config: HashConfig = field(default_factory=lambda: HashConfig(precision=32))

class SingleBloomFilter(PipelineStep):
    def __init__(
        self,
        output_folder: DataFolderLike,
        config: BloomFilterConfig,
        save_bloom_filter: bool = False,
        exclusion_writer: DiskWriter = None,
        language: str = Languages.english,
    ): ...
    def get_shingles(self, text: str) -> np.ndarray: ...
    def get_indexes(self, shingles: np.ndarray) -> list[list[int]]: ...
    def step(self, doc: Document) -> bool: ...
    def run(self, data: DocumentsPipeline, rank: int = 0, world_size: int = 1): ...

Import

from datatrove.pipeline.dedup.bloom_filter import SingleBloomFilter, BloomFilterConfig

I/O Contract

Inputs

Name	Type	Required	Description
output_folder	DataFolderLike	Yes	Output folder for saving the Bloom filter state (local or S3)
config	BloomFilterConfig	Yes	Configuration specifying filter size, k, n-gram size, and threshold
save_bloom_filter	bool	No	Whether to save the Bloom filter to disk after processing (default: False)
exclusion_writer	DiskWriter	No	Writer for saving excluded duplicate documents
language	str	No	Language for word tokenization (default: English)

Outputs

Name	Type	Description
Deduplicated documents	DocumentsPipeline	Documents with near-duplicates removed
bloom_filter.bloom	binary file	Persisted Bloom filter state (if save_bloom_filter is True)
Statistics	Stats	Counts of total, dropped, and forwarded documents

Usage Examples

Basic Usage

from datatrove.pipeline.dedup.bloom_filter import SingleBloomFilter, BloomFilterConfig

# Configure the Bloom filter
# m_bytes: 1GB filter, expected_elements: number of expected shingles
config = BloomFilterConfig(
    m_bytes=1_000_000_000,  # 1 GB
    expected_elements=100_000_000,
    n_grams=13,
    duplicate_threshold=0.8,
)

# Create the deduplication step
dedup_step = SingleBloomFilter(
    output_folder="output/bloom_dedup/",
    config=config,
    save_bloom_filter=True,
)

# Use in a pipeline
from datatrove.executor.local import LocalPipelineExecutor
executor = LocalPipelineExecutor(
    pipeline=[
        # ... reader step ...
        dedup_step,
        # ... writer step ...
    ],
    tasks=1,
    logging_dir="logs/bloom_dedup",
)
executor.run()

Related Pages

• Principle:Huggingface_Datatrove_Bloom_Filter_Deduplication