Implementation:Datajuicer Data juicer TopkSpecifiedFieldSelector

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data_Processing, Core
Last Updated	2026-02-14 16:00 GMT

Overview

Concrete tool for selecting top-k samples based on a specified field value provided by Data-Juicer.

Description

TopkSpecifiedFieldSelector is a selector operator that keeps the top-k samples from a dataset ranked by the numeric value of a specified field, supporting both ascending and descending sort orders. It extracts field values (supporting dot-separated multi-level keys), computes the selection count from top_ratio or topk (whichever results in fewer samples), then uses heapq.nlargest or heapq.nsmallest (based on the reverse flag) to efficiently select the top-ranked sample indices.

Usage

Use when you need to retain only the highest-scoring (or lowest-scoring) samples in a dataset according to a computed metric such as quality scores, text statistics, or any numeric field value.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/ops/selector/topk_specified_field_selector.py

Signature

@OPERATORS.register_module("topk_specified_field_selector")
class TopkSpecifiedFieldSelector(Selector):
    def __init__(
        self,
        field_key: str = "",
        top_ratio: Optional[Annotated[float, Field(ge=0, le=1)]] = None,
        topk: Optional[PositiveInt] = None,
        reverse: bool = True,
        *args,
        **kwargs,
    ):

Import

from data_juicer.ops.selector.topk_specified_field_selector import TopkSpecifiedFieldSelector

I/O Contract

Inputs

Name	Type	Required	Description
field_key	str	Yes	Dot-separated key path to the field used for ranking (e.g. "stats.text_len")
top_ratio	float (0-1)	No	Ratio of top samples to select from the dataset
topk	PositiveInt	No	Fixed number of top samples to select
reverse	bool	No	If True, sort descending (largest first); if False, sort ascending (smallest first). Default: True
dataset	Dataset	Yes	The input dataset to select from (passed to process())

Outputs

Name	Type	Description
dataset	Dataset	A subset of the input dataset containing only the top-k selected samples

Usage Examples

from data_juicer.ops.selector.topk_specified_field_selector import TopkSpecifiedFieldSelector

# Select top 100 samples by descending quality score
selector = TopkSpecifiedFieldSelector(
    field_key="stats.quality_score",
    topk=100,
    reverse=True
)
filtered_dataset = selector.process(dataset)

# Select top 10% of samples by ascending perplexity (lowest perplexity)
selector = TopkSpecifiedFieldSelector(
    field_key="stats.perplexity",
    top_ratio=0.1,
    reverse=False
)
filtered_dataset = selector.process(dataset)

Related Pages

• Environment:Datajuicer_Data_juicer_Python_Runtime_Environment