Implementation:Huggingface Datatrove BaseFilter

Knowledge Sources	Huggingface_Datatrove
Domains	Data Processing, Text Filtering
Last Updated	2026-02-14 17:00 GMT

Overview

BaseFilter is the abstract base class for all document filter pipeline steps in the Datatrove framework, providing the core logic for iterating over documents, applying filter decisions, tracking statistics, and optionally writing excluded documents to a separate output.

Description

BaseFilter extends both PipelineStep and Python's ABC (Abstract Base Class), establishing the contract that every concrete filter must implement the filter method. This method receives a single Document and returns either a boolean indicating whether to keep or drop the document, or a tuple of (False, reason_string) to drop with a specific explanatory reason. The helper function get_filter_result normalizes both return forms into a consistent (result, reason) tuple.

The class also supports batched filtering through the filter_batch method. When a subclass provides a custom filter_batch implementation and the batch_size parameter is set to a value greater than 1, documents are processed in batches rather than one at a time. If batch_size is greater than 1 but no custom filter_batch is implemented, a warning is emitted and the default implementation falls back to calling filter on each document individually.

The run method orchestrates the full pipeline execution: it groups incoming documents into batches, applies the filter, tracks statistics (total, forwarded, dropped, per-reason drops), and optionally writes dropped documents to an exclusion_writer of type DiskWriter. Dropped documents that include a reason string have that reason stored in their metadata["filter_reason"] field before being written.

Usage

Use BaseFilter as the parent class when implementing any custom document filter in a Datatrove pipeline. Subclasses only need to implement the filter method (and optionally filter_batch for batch-optimized processing). This class should never be instantiated directly.

Code Reference

Source Location

• Repository: Huggingface_Datatrove
• File: src/datatrove/pipeline/filters/base_filter.py
• Lines: 1-84

Signature

class BaseFilter(PipelineStep, ABC):
    type = "🔻 - FILTER"

    def __init__(self, exclusion_writer: DiskWriter = None, batch_size: int = 1):
        ...

    @abstractmethod
    def filter(self, doc: Document) -> bool | Tuple[bool, str]:
        ...

    def filter_batch(self, batch: List[Document]) -> List[bool | Tuple[bool, str]]:
        ...

    def run(self, data: DocumentsPipeline, rank: int = 0, world_size: int = 1) -> DocumentsPipeline:
        ...

Import

from datatrove.pipeline.filters.base_filter import BaseFilter

I/O Contract

Inputs

Name	Type	Required	Description
exclusion_writer	DiskWriter	No	Optional writer that saves dropped documents to disk
batch_size	int	No	Number of documents to process per batch (default: 1)

Outputs

Name	Type	Description
data	DocumentsPipeline (generator)	Yields only the documents that pass the filter

Usage Examples

Basic Usage

from datatrove.data import Document
from datatrove.pipeline.filters.base_filter import BaseFilter


class MinLengthFilter(BaseFilter):
    name = "Min Length Filter"

    def __init__(self, min_length: int = 100, exclusion_writer=None):
        super().__init__(exclusion_writer)
        self.min_length = min_length

    def filter(self, doc: Document) -> bool:
        return len(doc.text) >= self.min_length

Related Pages

• Principle:Huggingface_Datatrove_Document_Filtering_Framework