Implementation:Datajuicer Data juicer CharacterRepetitionFilter
| Knowledge Sources | |
|---|---|
| Domains | Data_Quality, Filtering |
| Last Updated | 2026-02-14 16:00 GMT |
Overview
Concrete tool for filtering data samples based on character-level n-gram repetition ratio provided by Data-Juicer.
Description
CharacterRepetitionFilter is a filter operator that keeps samples with character-level n-gram repetition ratio within a specific range. It calculates the repetition ratio based on the frequency of character n-grams in the text. The key metric char_rep_ratio is cached in the stats field. Samples are kept if their ratio falls within the configured min and max values. It extends the Filter base class and implements the two-phase compute_stats/process pattern.
Usage
Import this operator when you need to filter dataset samples based on the degree of character-level repetition in text. Configure it in your Data-Juicer YAML config or instantiate directly.
Code Reference
Source Location
- Repository: Datajuicer_Data_juicer
- File: data_juicer/ops/filter/character_repetition_filter.py
- Lines: 1-85
Signature
@OPERATORS.register_module("character_repetition_filter")
class CharacterRepetitionFilter(Filter):
def __init__(self, rep_len: PositiveInt = 10, min_ratio: float = 0.0, max_ratio: float = 0.5, *args, **kwargs):
...
Import
from data_juicer.ops.filter.character_repetition_filter import CharacterRepetitionFilter
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| rep_len | PositiveInt | No | Repetition length for character-level n-gram. Default: 10 |
| min_ratio | float | No | The minimum repetition ratio; samples below this are filtered out. Default: 0.0 |
| max_ratio | float | No | The maximum repetition ratio; samples above this are filtered out. Default: 0.5 |
Outputs
| Name | Type | Description |
|---|---|---|
| samples | Dict | Filtered samples with stats field updated (char_rep_ratio) |
Usage Examples
YAML Configuration
process:
- character_repetition_filter:
rep_len: 10
min_ratio: 0.0
max_ratio: 0.5
Python API
from data_juicer.ops.filter.character_repetition_filter import CharacterRepetitionFilter
op = CharacterRepetitionFilter(rep_len=10, min_ratio=0.0, max_ratio=0.5)
# Apply to dataset
result = dataset.process(op)