Implementation:Datajuicer Data juicer PhraseGroundingRecallFilter

Knowledge Sources	Datajuicer_Data_juicer
Domains	Multimodal Processing, Data Filtering, Phrase Grounding
Last Updated	2026-02-14 16:00 GMT

Overview

Filters multimodal samples based on phrase grounding recall, measuring how well noun phrases extracted from the text can be localized (grounded) in the associated images.

Description

PhraseGroundingRecallFilter is a multimodal filter operator that ensures text descriptions actually correspond to visible content in images at the phrase level. It extracts noun phrases from text using NLTK POS tagging and a regex grammar adapted from the GLIP project, then uses a Hugging Face Owl-ViT model to detect bounding boxes for each noun phrase in the image.

The recall is computed as the fraction of noun phrases that are successfully grounded (i.e., detected with IoU above a configurable threshold). Samples are kept if the recall falls within a specified min/max range. The operator supports:

Multiple images per text chunk with configurable reduce modes (avg, max, min)
any or all filtering strategies across images
Horizontal and vertical image flipping
NMS-like post-processing to suppress overlapping bounding boxes
Large area ratio filtering to discard predictions covering most of the image
Confidence score thresholding

Helper functions find_noun_phrases, remove_punctuation, and run_ner implement the NER extraction pipeline adapted from Microsoft GLIP.

Usage

Use this filter when building multimodal datasets that require strong text-image alignment at the phrase level. It is critical for training grounding and referring expression models where each mentioned entity must be visible in the corresponding image.

Code Reference

Source Location

Repository: Datajuicer_Data_juicer
File: data_juicer/ops/filter/phrase_grounding_recall_filter.py
Lines: 1-326

Signature

class PhraseGroundingRecallFilter(Filter):
    _accelerator = "cuda"

    def __init__(
        self,
        hf_owlvit: str = "google/owlvit-base-patch32",
        trust_remote_code: bool = False,
        min_recall: float = 0.1,
        max_recall: float = 1.0,
        horizontal_flip: bool = False,
        vertical_flip: bool = False,
        any_or_all: str = "any",
        reduce_mode: str = "avg",
        iou_thr: float = 0.5,
        large_area_ratio_thr: float = 0.95,
        conf_thr: float = 0.0,
        *args, **kwargs,
    ):

Import

from data_juicer.ops.filter.phrase_grounding_recall_filter import PhraseGroundingRecallFilter

I/O Contract

Inputs

Name	Type	Required	Description
hf_owlvit	str	No	Owl-ViT model name on HuggingFace for phrase grounding. Default: "google/owlvit-base-patch32"
trust_remote_code	bool	No	Whether to trust remote code of HF models. Default: False
min_recall	float	No	Minimum phrase grounding recall to keep samples. Default: 0.1
max_recall	float	No	Maximum phrase grounding recall to keep samples. Default: 1.0
horizontal_flip	bool	No	Flip image horizontally before grounding. Default: False
vertical_flip	bool	No	Flip image vertically before grounding. Default: False
any_or_all	str	No	Keep strategy across images: "any" or "all". Default: "any"
reduce_mode	str	No	Reduce mode for multiple images per chunk: "avg", "max", or "min". Default: "avg"
iou_thr	float	No	IoU threshold for NMS-like post-processing. Default: 0.5
large_area_ratio_thr	float	No	Area ratio threshold for filtering large bboxes. Default: 0.95
conf_thr	float	No	Confidence score threshold for bbox filtering. Default: 0.0

Outputs

Name	Type	Description
sample[Fields.stats][StatsKeys.phrase_grounding_recall]	list[float]	Recall values per text chunk, stored in sample stats
bool	bool	True/False indicating whether to keep the sample

Usage Examples

# Basic usage with default Owl-ViT model
filter_op = PhraseGroundingRecallFilter(
    hf_owlvit="google/owlvit-base-patch32",
    min_recall=0.3,
    max_recall=1.0,
    any_or_all="any",
    reduce_mode="avg",
)

# With stricter filtering
filter_op = PhraseGroundingRecallFilter(
    min_recall=0.5,
    max_recall=1.0,
    any_or_all="all",
    iou_thr=0.3,
    conf_thr=0.1,
)

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment