Implementation:Huggingface Datatrove BaseFilter
| Knowledge Sources | |
|---|---|
| Domains | Data Processing, Text Filtering |
| Last Updated | 2026-02-14 17:00 GMT |
Overview
BaseFilter is the abstract base class for all document filter pipeline steps in the Datatrove framework, providing the core logic for iterating over documents, applying filter decisions, tracking statistics, and optionally writing excluded documents to a separate output.
Description
BaseFilter extends both PipelineStep and Python's ABC (Abstract Base Class), establishing the contract that every concrete filter must implement the filter method. This method receives a single Document and returns either a boolean indicating whether to keep or drop the document, or a tuple of (False, reason_string) to drop with a specific explanatory reason. The helper function get_filter_result normalizes both return forms into a consistent (result, reason) tuple.
The class also supports batched filtering through the filter_batch method. When a subclass provides a custom filter_batch implementation and the batch_size parameter is set to a value greater than 1, documents are processed in batches rather than one at a time. If batch_size is greater than 1 but no custom filter_batch is implemented, a warning is emitted and the default implementation falls back to calling filter on each document individually.
The run method orchestrates the full pipeline execution: it groups incoming documents into batches, applies the filter, tracks statistics (total, forwarded, dropped, per-reason drops), and optionally writes dropped documents to an exclusion_writer of type DiskWriter. Dropped documents that include a reason string have that reason stored in their metadata["filter_reason"] field before being written.
Usage
Use BaseFilter as the parent class when implementing any custom document filter in a Datatrove pipeline. Subclasses only need to implement the filter method (and optionally filter_batch for batch-optimized processing). This class should never be instantiated directly.
Code Reference
Source Location
- Repository: Huggingface_Datatrove
- File: src/datatrove/pipeline/filters/base_filter.py
- Lines: 1-84
Signature
class BaseFilter(PipelineStep, ABC):
type = "🔻 - FILTER"
def __init__(self, exclusion_writer: DiskWriter = None, batch_size: int = 1):
...
@abstractmethod
def filter(self, doc: Document) -> bool | Tuple[bool, str]:
...
def filter_batch(self, batch: List[Document]) -> List[bool | Tuple[bool, str]]:
...
def run(self, data: DocumentsPipeline, rank: int = 0, world_size: int = 1) -> DocumentsPipeline:
...
Import
from datatrove.pipeline.filters.base_filter import BaseFilter
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| exclusion_writer | DiskWriter | No | Optional writer that saves dropped documents to disk |
| batch_size | int | No | Number of documents to process per batch (default: 1) |
Outputs
| Name | Type | Description |
|---|---|---|
| data | DocumentsPipeline (generator) | Yields only the documents that pass the filter |
Usage Examples
Basic Usage
from datatrove.data import Document
from datatrove.pipeline.filters.base_filter import BaseFilter
class MinLengthFilter(BaseFilter):
name = "Min Length Filter"
def __init__(self, min_length: int = 100, exclusion_writer=None):
super().__init__(exclusion_writer)
self.min_length = min_length
def filter(self, doc: Document) -> bool:
return len(doc.text) >= self.min_length