Implementation:Datajuicer Data juicer InContextInfluenceFilter

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data_Quality, Filtering
Last Updated	2026-02-14 16:00 GMT

Overview

Concrete tool for filtering data samples based on their in-context influence on a validation set provided by Data-Juicer.

Description

InContextInfluenceFilter is a filter operator that keeps texts based on their in-context influence on a validation set. It calculates the influence by comparing perplexities with and without the sample as context. If valid_as_demo is True, the score is L(A|Q) / L(A|task_desc, Q_v, A_v, Q); otherwise it is L(A_v|Q) / L(A_v|task_desc, Q, A, Q_v). The operator retains samples whose in-context influence score is within the specified range. It extends LLMPerplexityFilter (which extends Filter) and implements the two-phase compute_stats/process pattern.

Usage

Import this operator when you need to filter dataset samples based on their in-context influence on downstream validation tasks. The validation set must be prepared before applying the filter. Configure it in your Data-Juicer YAML config or instantiate directly.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/ops/filter/in_context_influence_filter.py
• Lines: 1-119

Signature

@OPERATORS.register_module("in_context_influence_filter")
@ATTRIBUTION_FILTERS.register_module("in_context_influence_filter")
class InContextInfluenceFilter(LLMPerplexityFilter):
    def __init__(
        self,
        valid_dataset: Optional[List[Dict]] = None,
        task_desc: str = None,
        valid_as_demo: bool = False,
        n_shot: Optional[int] = None,
        *args,
        **kwargs,
    ):
        ...

Import

from data_juicer.ops.filter.in_context_influence_filter import InContextInfluenceFilter

I/O Contract

Inputs

Name	Type	Required	Description
valid_dataset	Optional[List[Dict]]	No	The dataset to use for validation. If None, prepare_valid_feature must be called manually. Default: None
task_desc	str	No	The description of the validation task. Default: None
valid_as_demo	bool	No	Q) / L(A|task_desc, Q_v, A_v, Q); if False, score = L(A_v|Q) / L(A_v|task_desc, Q, A, Q_v). Default: False
n_shot	Optional[int]	No	The number of shots in validation. Default: None

Outputs

Name	Type	Description
samples	Dict	Filtered samples with stats field updated (in_context_influence)

Usage Examples

YAML Configuration

process:
  - in_context_influence_filter:
      valid_as_demo: false
      min_score: 1.0
      max_score: 100.0

Python API

from data_juicer.ops.filter.in_context_influence_filter import InContextInfluenceFilter

op = InContextInfluenceFilter(valid_dataset=valid_data, task_desc="Summarization task")
# Apply to dataset
result = dataset.process(op)

Related Pages

• Environment:Datajuicer_Data_juicer_Python_Runtime_Environment