Implementation:Datajuicer Data juicer SpecifiedNumericFieldFilter
| Knowledge Sources | |
|---|---|
| Domains | Data_Quality, Filtering |
| Last Updated | 2026-02-14 16:00 GMT |
Overview
Concrete tool for filtering data samples based on a specified numeric field value provided by Data-Juicer.
Description
SpecifiedNumericFieldFilter is a filter operator that keeps samples where a specified numeric field value falls within a given min/max range. It extends Filter and uses the two-phase compute_stats/process pattern. In compute_stats_single, it traverses the sample dictionary using a dot-separated field_key to extract the numeric field value and copies it to stats. In process_single, it validates the value is numeric using the helper is_number() function, then checks if it falls within [min_value, max_value]. Complements SpecifiedFieldFilter by handling numeric range-based filtering on arbitrary sample fields.
Usage
Import when filtering based on numeric field ranges. Configure in YAML or Python.
Code Reference
Source Location
- Repository: Datajuicer_Data_juicer
- File: data_juicer/ops/filter/specified_numeric_field_filter.py
Signature
@OPERATORS.register_module("specified_numeric_field_filter")
class SpecifiedNumericFieldFilter(Filter):
def __init__(self, field_key: str = "", min_value: float = -sys.maxsize, max_value: float = sys.maxsize, *args, **kwargs):
Import
from data_juicer.ops.filter.specified_numeric_field_filter import SpecifiedNumericFieldFilter
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| field_key | str | No | Dot-separated key path to the target numeric field (default: "") |
| min_value | float | No | Minimum value threshold (default: -sys.maxsize) |
| max_value | float | No | Maximum value threshold (default: sys.maxsize) |
Outputs
| Name | Type | Description |
|---|---|---|
| samples | Dict | Filtered samples based on numeric field range |
Usage Examples
YAML Configuration
process:
- specified_numeric_field_filter:
field_key: "meta.quality_score"
min_value: 0.5
max_value: 1.0
Python API
from data_juicer.ops.filter.specified_numeric_field_filter import SpecifiedNumericFieldFilter
op = SpecifiedNumericFieldFilter(field_key="meta.quality_score", min_value=0.5, max_value=1.0)