Implementation:Datajuicer Data juicer LanguageIDScoreFilter
| Knowledge Sources | |
|---|---|
| Domains | Data_Quality, Filtering |
| Last Updated | 2026-02-14 16:00 GMT |
Overview
Concrete tool for filtering data samples based on language identification confidence score provided by Data-Juicer.
Description
LanguageIDScoreFilter is a filter operator that keeps samples in a specific language with a confidence score above a threshold. It uses a FastText model to identify the language of each sample. Samples are kept if they are in the specified language(s) and have a confidence score at or above the minimum. If no specific language is provided, it only filters based on the confidence score. The language ID and score are stored under the lang and lang_score stats keys. It extends the Filter base class and implements the two-phase compute_stats/process pattern.
Usage
Import this operator when you need to filter dataset samples based on the detected language and confidence of language identification. Configure it in your Data-Juicer YAML config or instantiate directly.
Code Reference
Source Location
- Repository: Datajuicer_Data_juicer
- File: data_juicer/ops/filter/language_id_score_filter.py
- Lines: 1-73
Signature
@OPERATORS.register_module("language_id_score_filter")
class LanguageIDScoreFilter(Filter):
def __init__(self, lang: Union[str, List[str]] = "", min_score: float = 0.8, *args, **kwargs):
...
Import
from data_juicer.ops.filter.language_id_score_filter import LanguageIDScoreFilter
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| lang | Union[str, List[str]] | No | Language(s) to keep samples in. Empty string means no language constraint. Default: "" |
| min_score | float | No | The minimum language identification confidence score to keep samples. Default: 0.8 |
Outputs
| Name | Type | Description |
|---|---|---|
| samples | Dict | Filtered samples with stats field updated (lang, lang_score) |
Usage Examples
YAML Configuration
process:
- language_id_score_filter:
lang: "en"
min_score: 0.8
Python API
from data_juicer.ops.filter.language_id_score_filter import LanguageIDScoreFilter
op = LanguageIDScoreFilter(lang="en", min_score=0.8)
# Apply to dataset
result = dataset.process(op)