Implementation:Datajuicer Data juicer VideoFramesTextSimilarityFilter
| Knowledge Sources | |
|---|---|
| Domains | Data_Quality, Filtering |
| Last Updated | 2026-02-14 16:00 GMT |
Overview
Concrete tool for filtering data samples based on video frame-text similarity provided by Data-Juicer.
Description
VideoFramesTextSimilarityFilter is a filter operator that keeps samples where the CLIP-computed similarity between sampled video frames and their associated text falls within a specified range. It extends Filter and uses the two-phase compute_stats/process pattern. It extracts frames using either 'all_keyframes' or 'uniform' sampling, optionally flips them, and uses a HuggingFace CLIP model (default: openai/clip-vit-base-patch32) to compute cosine similarity with the text. Per-video scores are reduced via 'avg', 'max', or 'min' across frames. Results are cached under video_frames_text_similarity. Supports CUDA acceleration and operator fusion. Ensures video-text alignment in multimodal datasets.
Usage
Import when filtering based on video frame-text alignment. Configure in YAML or Python.
Code Reference
Source Location
- Repository: Datajuicer_Data_juicer
- File: data_juicer/ops/filter/video_frames_text_similarity_filter.py
Signature
@OPERATORS.register_module("video_frames_text_similarity_filter")
class VideoFramesTextSimilarityFilter(Filter):
def __init__(self, hf_clip="openai/clip-vit-base-patch32", trust_remote_code=False, min_score: float = 0.1, max_score: float = 1.0, frame_sampling_method: str = "all_keyframes", frame_num: PositiveInt = 3, horizontal_flip: bool = False, vertical_flip: bool = False, any_or_all: str = "any", reduce_mode: str = "avg", *args, **kwargs):
Import
from data_juicer.ops.filter.video_frames_text_similarity_filter import VideoFramesTextSimilarityFilter
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| hf_clip | str | No | CLIP model name on HuggingFace (default: "openai/clip-vit-base-patch32") |
| min_score | float | No | Minimum similarity score (default: 0.1) |
| max_score | float | No | Maximum similarity score (default: 1.0) |
| frame_sampling_method | str | No | Frame sampling: "all_keyframes" or "uniform" (default: "all_keyframes") |
| frame_num | PositiveInt | No | Number of frames for uniform sampling (default: 3) |
| horizontal_flip | bool | No | Flip frames horizontally (default: False) |
| vertical_flip | bool | No | Flip frames vertically (default: False) |
| any_or_all | str | No | Keep strategy: "any" or "all" (default: "any") |
| reduce_mode | str | No | Score reduction: "avg", "max", or "min" (default: "avg") |
Outputs
| Name | Type | Description |
|---|---|---|
| samples | Dict | Filtered samples with video_frames_text_similarity stat computed |
Usage Examples
YAML Configuration
process:
- video_frames_text_similarity_filter:
min_score: 0.1
max_score: 1.0
frame_sampling_method: all_keyframes
Python API
from data_juicer.ops.filter.video_frames_text_similarity_filter import VideoFramesTextSimilarityFilter
op = VideoFramesTextSimilarityFilter(min_score=0.1, max_score=1.0)