Implementation:Datajuicer Data juicer SuffixFilter

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data_Quality, Filtering
Last Updated	2026-02-14 16:00 GMT

Overview

Concrete tool for filtering data samples based on file suffix provided by Data-Juicer.

Description

SuffixFilter is a filter operator that keeps samples whose file suffix (extension) matches a provided list of allowed suffixes. It extends Filter and uses the two-phase compute_stats/process pattern. The compute_stats_single method is a no-op since no statistics need to be computed. In process_single, it checks whether the sample's suffix field is in the allowed list. Supports reversed_range to invert the filter. Registered as a NON_STATS_FILTER since it does not compute statistical metrics.

Usage

Import when filtering based on file suffix/extension. Configure in YAML or Python.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/ops/filter/suffix_filter.py

Signature

@OPERATORS.register_module("suffix_filter")
class SuffixFilter(Filter):
    def __init__(self, suffixes: Union[str, List[str]] = [], *args, **kwargs):

Import

from data_juicer.ops.filter.suffix_filter import SuffixFilter

I/O Contract

Inputs

Name	Type	Required	Description
suffixes	Union[str, List[str]]	No	Suffixes to keep (e.g., '.txt', 'pdf', ['txt', '.pdf', 'docx'])

Outputs

Name	Type	Description
samples	Dict	Filtered samples matching the specified suffixes

Usage Examples

YAML Configuration

process:
  - suffix_filter:
      suffixes: ['.txt', '.pdf', '.docx']

Python API

from data_juicer.ops.filter.suffix_filter import SuffixFilter
op = SuffixFilter(suffixes=['.txt', '.pdf', '.docx'])

Related Pages

• Environment:Datajuicer_Data_juicer_Python_Runtime_Environment