Implementation:Datajuicer Data juicer WordsNumFilter
| Knowledge Sources | |
|---|---|
| Domains | Data_Quality, Filtering |
| Last Updated | 2026-02-14 16:00 GMT |
Overview
Concrete tool for filtering data samples based on word count provided by Data-Juicer.
Description
WordsNumFilter is a filter operator that keeps samples whose word count falls within a specified range. It extends Filter and uses the two-phase compute_stats/process pattern. It tokenizes text using either a SentencePiece model (when tokenization=True) or whitespace splitting, with optional word refinement (stripping special characters). Supports operator fusion via INTER_WORDS for reusing tokenized words across operators. Caches the word count under num_words and checks against [min_num, max_num] thresholds in batched processing. Fundamental text length filter operating at the word level rather than character level, providing a more linguistically meaningful measure of content length.
Usage
Import when filtering based on word count. Configure in YAML or Python.
Code Reference
Source Location
- Repository: Datajuicer_Data_juicer
- File: data_juicer/ops/filter/words_num_filter.py
Signature
@OPERATORS.register_module("words_num_filter")
class WordsNumFilter(Filter):
def __init__(self, lang: str = "en", tokenization: bool = False, min_num: int = 10, max_num: int = sys.maxsize, *args, **kwargs):
Import
from data_juicer.ops.filter.words_num_filter import WordsNumFilter
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| lang | str | No | Language of the text (default: "en") |
| tokenization | bool | No | Whether to use model tokenizer (default: False) |
| min_num | int | No | Minimum word count to keep samples (default: 10) |
| max_num | int | No | Maximum word count to keep samples (default: sys.maxsize) |
Outputs
| Name | Type | Description |
|---|---|---|
| samples | Dict | Filtered samples with num_words stat computed |
Usage Examples
YAML Configuration
process:
- words_num_filter:
lang: en
min_num: 10
max_num: 10000
Python API
from data_juicer.ops.filter.words_num_filter import WordsNumFilter
op = WordsNumFilter(min_num=10, max_num=10000)