Implementation:Datajuicer Data juicer SpecialCharactersFilter

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data_Quality, Filtering
Last Updated	2026-02-14 16:00 GMT

Overview

Concrete tool for filtering data samples based on special character ratio provided by Data-Juicer.

Description

SpecialCharactersFilter is a filter operator that keeps samples where the ratio of special characters to total characters falls within a specified range. It extends Filter and uses the two-phase compute_stats/process pattern. In compute_stats_batched, it counts characters that appear in the predefined SPECIAL_CHARACTERS set, divides by total text length, and caches the result under special_char_ratio. Filters out text with excessive special characters such as HTML artifacts, encoding errors, or markup-heavy content. Adapted from HuggingFace's text data filtering methodology.

Usage

Import when filtering based on special character ratio. Configure in YAML or Python.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/ops/filter/special_characters_filter.py

Signature

@OPERATORS.register_module("special_characters_filter")
class SpecialCharactersFilter(Filter):
    def __init__(self, min_ratio: float = 0.0, max_ratio: float = 0.25, *args, **kwargs):

Import

from data_juicer.ops.filter.special_characters_filter import SpecialCharactersFilter

I/O Contract

Inputs

Name	Type	Required	Description
min_ratio	float	No	Minimum special character ratio (default: 0.0)
max_ratio	float	No	Maximum special character ratio (default: 0.25)

Outputs

Name	Type	Description
samples	Dict	Filtered samples with special_char_ratio stat computed

Usage Examples

YAML Configuration

process:
  - special_characters_filter:
      min_ratio: 0.0
      max_ratio: 0.25

Python API

from data_juicer.ops.filter.special_characters_filter import SpecialCharactersFilter
op = SpecialCharactersFilter(min_ratio=0.0, max_ratio=0.25)

Related Pages

• Environment:Datajuicer_Data_juicer_Python_Runtime_Environment