Implementation:Datajuicer Data juicer TextLengthFilter

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data_Quality, Filtering
Last Updated	2026-02-14 16:00 GMT

Overview

Concrete tool for filtering data samples based on total text length provided by Data-Juicer.

Description

TextLengthFilter is a filter operator that keeps samples whose total text length (character count) falls within a specified range. It extends Filter and uses the two-phase compute_stats/process pattern. In compute_stats_batched, it computes len(text) for each sample and caches it under the text_len stats key. The process_batched method checks this value against [min_len, max_len] thresholds. One of the most fundamental text quality filters, removing samples that are too short (e.g., empty or trivial entries) or too long (e.g., data dumps) for effective LLM training.

Usage

Import when filtering based on text character length. Configure in YAML or Python.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/ops/filter/text_length_filter.py

Signature

@OPERATORS.register_module("text_length_filter")
class TextLengthFilter(Filter):
    def __init__(self, min_len: int = 10, max_len: int = sys.maxsize, *args, **kwargs):

Import

from data_juicer.ops.filter.text_length_filter import TextLengthFilter

I/O Contract

Inputs

Name	Type	Required	Description
min_len	int	No	Minimum text length in characters (default: 10)
max_len	int	No	Maximum text length in characters (default: sys.maxsize)

Outputs

Name	Type	Description
samples	Dict	Filtered samples with text_len stat computed

Usage Examples

YAML Configuration

process:
  - text_length_filter:
      min_len: 100
      max_len: 10000

Python API

from data_juicer.ops.filter.text_length_filter import TextLengthFilter
op = TextLengthFilter(min_len=100, max_len=10000)

Related Pages

• Environment:Datajuicer_Data_juicer_Python_Runtime_Environment