Implementation:Datajuicer Data juicer TextPairSimilarityFilter

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data_Quality, Filtering
Last Updated	2026-02-14 16:00 GMT

Overview

Concrete tool for filtering data samples based on text pair similarity provided by Data-Juicer.

Description

TextPairSimilarityFilter is a filter operator that keeps text pairs whose cosine similarity between CLIP text embeddings falls within a specified range. It extends Filter and uses the two-phase compute_stats/process pattern. It uses a HuggingFace CLIP model (default: openai/clip-vit-base-patch32) to embed both texts from a pair (using text_key and text_key_second), computes cosine similarity between the embeddings, and caches the score under text_pair_similarity. Requires text_key_second to be set. Supports 'any'/'all' strategy and CUDA acceleration.

Usage

Import when filtering based on text pair similarity. Configure in YAML or Python.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/ops/filter/text_pair_similarity_filter.py

Signature

@OPERATORS.register_module("text_pair_similarity_filter")
class TextPairSimilarityFilter(Filter):
    def __init__(self, hf_clip="openai/clip-vit-base-patch32", trust_remote_code=False, min_score: ClosedUnitInterval = 0.1, max_score: ClosedUnitInterval = 1.0, text_key_second=None, any_or_all: str = "any", *args, **kwargs):

Import

from data_juicer.ops.filter.text_pair_similarity_filter import TextPairSimilarityFilter

I/O Contract

Inputs

Name	Type	Required	Description
hf_clip	str	No	CLIP model name on HuggingFace (default: "openai/clip-vit-base-patch32")
trust_remote_code	bool	No	Whether to trust remote code of HF models (default: False)
min_score	ClosedUnitInterval	No	Minimum similarity to keep samples (default: 0.1)
max_score	ClosedUnitInterval	No	Maximum similarity to keep samples (default: 1.0)
text_key_second	str	Yes	Key for the second text in the pair
any_or_all	str	No	Keep strategy: "any" or "all" (default: "any")

Outputs

Name	Type	Description
samples	Dict	Filtered samples with text_pair_similarity stat computed

Usage Examples

YAML Configuration

process:
  - text_pair_similarity_filter:
      min_score: 0.1
      max_score: 1.0
      text_key_second: "text2"

Python API

from data_juicer.ops.filter.text_pair_similarity_filter import TextPairSimilarityFilter
op = TextPairSimilarityFilter(min_score=0.1, text_key_second="text2")

Related Pages

• Environment:Datajuicer_Data_juicer_Python_Runtime_Environment