Implementation:Datajuicer Data juicer ImageTextSimilarityFilter

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data_Quality, Filtering
Last Updated	2026-02-14 16:00 GMT

Overview

Concrete tool for filtering data samples based on image-text similarity provided by Data-Juicer.

Description

ImageTextSimilarityFilter is a filter operator that keeps samples with image-text similarity within a specified range. It uses a HuggingFace CLIP model (default: openai/clip-vit-base-patch32) to compute cosine similarity between image and text embeddings. The similarity score is normalized by dividing logits_per_text by 100. Scores can be reduced using 'avg', 'max', or 'min' modes when one text corresponds to multiple images. The operator supports horizontal and vertical image flipping, CUDA acceleration, and 'any'/'all' strategies. The key metric image_text_similarity is cached in the stats field. It extends the Filter base class and implements the two-phase compute_stats/process pattern.

Usage

Import this operator when you need to filter dataset samples based on semantic similarity between images and text. Configure it in your Data-Juicer YAML config or instantiate directly.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/ops/filter/image_text_similarity_filter.py
• Lines: 1-165

Signature

@OPERATORS.register_module("image_text_similarity_filter")
@LOADED_IMAGES.register_module("image_text_similarity_filter")
class ImageTextSimilarityFilter(Filter):
    def __init__(
        self,
        hf_clip: str = "openai/clip-vit-base-patch32",
        trust_remote_code: bool = False,
        min_score: float = 0.1,
        max_score: float = 1.0,
        horizontal_flip: bool = False,
        vertical_flip: bool = False,
        any_or_all: str = "any",
        reduce_mode: str = "avg",
        *args,
        **kwargs,
    ):
        ...

Import

from data_juicer.ops.filter.image_text_similarity_filter import ImageTextSimilarityFilter

I/O Contract

Inputs

Name	Type	Required	Description
hf_clip	str	No	CLIP model name on HuggingFace for computing similarity. Default: "openai/clip-vit-base-patch32"
trust_remote_code	bool	No	Whether to trust remote code of HF models. Default: False
min_score	float	No	The minimum similarity score to keep samples. Default: 0.1
max_score	float	No	The maximum similarity score to keep samples. Default: 1.0
horizontal_flip	bool	No	Flip image horizontally (left to right). Default: False
vertical_flip	bool	No	Flip image vertically (top to bottom). Default: False
any_or_all	str	No	Keep strategy: 'any' or 'all' across images. Default: "any"
reduce_mode	str	No	Reduce mode for multiple images per chunk: 'avg', 'max', or 'min'. Default: "avg"

Outputs

Name	Type	Description
samples	Dict	Filtered samples with stats field updated (image_text_similarity)

Usage Examples

YAML Configuration

process:
  - image_text_similarity_filter:
      hf_clip: "openai/clip-vit-base-patch32"
      min_score: 0.1
      max_score: 1.0
      any_or_all: "any"
      reduce_mode: "avg"

Python API

from data_juicer.ops.filter.image_text_similarity_filter import ImageTextSimilarityFilter

op = ImageTextSimilarityFilter(min_score=0.1, max_score=1.0)
# Apply to dataset
result = dataset.process(op)

Related Pages

• Environment:Datajuicer_Data_juicer_Python_Runtime_Environment