Implementation:Datajuicer Data juicer VideoCaptioningFromSummarizerMapper

Knowledge Sources	Datajuicer_Data_juicer
Domains	Data_Processing, Mapping
Last Updated	2026-02-14 16:00 GMT

Overview

Concrete tool for generating video captions by summarizing multiple text sources provided by Data-Juicer.

Description

VideoCaptioningFromSummarizerMapper is a mapper operator that generates comprehensive video captions by combining and summarizing multiple text sources -- captions from video, audio, and frames, plus tags from audio and frames -- using a HuggingFace summarization model. It orchestrates multiple subordinate captioning and tagging operators, collects their outputs, keeps the top-N most frequent frame tags, compiles all information, and feeds it to a summarizer model to produce a coherent final caption.

Usage

Use when you need the richest possible video descriptions by fusing multimodal information from multiple specialized sub-operators into a single summarized caption.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/ops/mapper/video_captioning_from_summarizer_mapper.py

Signature

@OPERATORS.register_module("video_captioning_from_summarizer_mapper")
class VideoCaptioningFromSummarizerMapper(Mapper):
    def __init__(self, hf_summarizer: str = None, trust_remote_code: bool = False, consider_video_caption_from_video: bool = True, consider_video_caption_from_audio: bool = True, consider_video_caption_from_frames: bool = True, consider_video_tags_from_audio: bool = True, consider_video_tags_from_frames: bool = True, vid_cap_from_vid_args: Optional[Dict] = None, vid_cap_from_frm_args: Optional[Dict] = None, vid_tag_from_aud_args: Optional[Dict] = None, vid_tag_from_frm_args: Optional[Dict] = None, keep_tag_num: PositiveInt = 5, keep_original_sample: bool = True, *args, **kwargs):

Import

from data_juicer.ops.mapper.video_captioning_from_summarizer_mapper import VideoCaptioningFromSummarizerMapper

I/O Contract

Inputs

Name	Type	Required	Description
hf_summarizer	str	No	HuggingFace summarizer model id (default: flan-t5-large-finetuned-openai-summarize_from_feedback)
trust_remote_code	bool	No	Whether to trust remote code of HF models (default: False)
consider_video_caption_from_video	bool	No	Whether to include caption from video directly (default: True)
consider_video_caption_from_audio	bool	No	Whether to include caption from audio streams (default: True)
consider_video_caption_from_frames	bool	No	Whether to include caption from sampled frames (default: True)
consider_video_tags_from_audio	bool	No	Whether to include tags from audio streams (default: True)
consider_video_tags_from_frames	bool	No	Whether to include tags from sampled frames (default: True)
vid_cap_from_vid_args	Dict	No	Arguments for video captioning from video operator (default: None)
vid_cap_from_frm_args	Dict	No	Arguments for video captioning from frames operator (default: None)
vid_tag_from_aud_args	Dict	No	Arguments for video tagging from audio operator (default: None)
vid_tag_from_frm_args	Dict	No	Arguments for video tagging from frames operator (default: None)
keep_tag_num	PositiveInt	No	Max number of most frequent tags to keep (default: 5)
keep_original_sample	bool	No	Whether to keep the original sample (default: True)

Outputs

Name	Type	Description
samples	Dict	Transformed samples with summarized video captions

Usage Examples

process:
  - video_captioning_from_summarizer_mapper:
      consider_video_caption_from_video: true
      consider_video_caption_from_audio: true
      keep_tag_num: 5

Related Pages

• Environment:Datajuicer_Data_juicer_Python_Runtime_Environment