Implementation:Datajuicer Data juicer PerplexityFilter

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data_Quality, Filtering
Last Updated	2026-02-14 16:00 GMT

Overview

Concrete tool for filtering data samples based on perplexity score provided by Data-Juicer.

Description

PerplexityFilter is a filter operator that keeps samples whose perplexity scores fall within a specified range. It extends Filter and uses the two-phase compute_stats/process pattern. It tokenizes text using a SentencePiece model, then computes character-based perplexity via a KenLM n-gram language model by accumulating log-probabilities across lines and exponentiating. The perplexity is cached under the perplexity stats key. Supports operator fusion via INTER_WORDS for reusing tokenized words across multiple operators. Adapted from HuggingFace's text data filtering pipeline.

Usage

Import when filtering based on text perplexity. Configure in YAML or Python.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/ops/filter/perplexity_filter.py

Signature

@OPERATORS.register_module("perplexity_filter")
class PerplexityFilter(Filter):
    def __init__(self, lang: str = "en", min_ppl: float = 0, max_ppl: float = 1500, *args, **kwargs):

Import

from data_juicer.ops.filter.perplexity_filter import PerplexityFilter

I/O Contract

Inputs

Name	Type	Required	Description
lang	str	No	Language for perplexity computation (default: "en")
min_ppl	float	No	Minimum perplexity threshold (default: 0)
max_ppl	float	No	Maximum perplexity threshold (default: 1500)

Outputs

Name	Type	Description
samples	Dict	Filtered samples with perplexity stat computed

Usage Examples

YAML Configuration

process:
  - perplexity_filter:
      lang: en
      max_ppl: 1500

Python API

from data_juicer.ops.filter.perplexity_filter import PerplexityFilter
op = PerplexityFilter(lang="en", max_ppl=1500)

Related Pages

• Environment:Datajuicer_Data_juicer_Python_Runtime_Environment