Implementation:Datajuicer Data juicer Video Utils

Knowledge Sources	Datajuicer_Data_juicer
Domains	Video Processing, Multimodal Processing
Last Updated	2026-02-14 16:00 GMT

Overview

Comprehensive video processing utility module providing a backend-agnostic API with multiple video reader implementations (PyAV, FFmpeg, Decord), metadata extraction, frame sampling, keyframe extraction, and clip management.

Description

The video_utils module provides core video processing infrastructure used by all video-related operators, designed around an abstract base class pattern with three concrete backends:

Data Classes:

• VideoMetadata -- Dataclass storing height, width, fps, num_frames, and duration.
• Frames -- Container (via attrs) holding frame arrays, indices, and pts timestamps.
• Clip -- Container for video clip data including source video path, time span, optional output path, encoded bytes, and numpy frame arrays.

Abstract Base Class:

• VideoReader -- Defines the interface with methods:
  • get_metadata() -- Returns VideoMetadata.
  • extract_frames(start_time, end_time) -- Yields numpy arrays for frames in a time range.
  • extract_keyframes(start_time, end_time) -- Returns Frames object with keyframe data.
  • extract_clip(start_time, end_time, output_path, to_numpy) -- Returns a Clip object.
  • close() -- Releases resources.
  • Supports context manager protocol (__enter__/__exit__).
  • check_time_span -- Validates time parameters.

Concrete Implementations:

AVReader (PyAV backend):

• Uses the av library for decoding.
• Supports seeking to specific timestamps and frame-level PTS-based filtering.
• Keyframe extraction uses codec context skip_frame = "NONKEY".
• Clip extraction delegates to cut_video_by_seconds from mm_utils or extracts frames directly.

FFmpegReader (FFmpeg subprocess backend):

• Uses FFmpeg/FFprobe via subprocess for decoding.
• Handles bytes and file-like inputs via temporary files.
• Frame extraction pipes raw video data from ffmpeg stdout.
• Keyframe extraction uses ffmpeg's select=eq(pict_type,I) filter with concurrent stderr parsing in a separate thread.
• Clip extraction uses stream copy (-c copy) for fast clipping without re-encoding.
• Proper process cleanup with timeout-based termination.

DecordReader (Decord backend):

• Uses the decord library for GPU-friendly batch decoding.
• Frame extraction via get_batch() for efficient batch access.
• Keyframe extraction via get_key_indices().
• Currently supports only numpy output (no encoded clip output).

Factory Function:

• create_video_reader -- Selects available backend automatically or uses a specified backend ("ffmpeg", "decord", "av").

Usage

Use this module for all video reading, frame extraction, and clip management operations. The factory function create_video_reader provides the simplest entry point, automatically selecting the best available backend.

Code Reference

Source Location

• Repository: Datajuicer_Data_juicer
• File: data_juicer/utils/video_utils.py

Signature

@dataclass
class VideoMetadata:
    height: int | None = None
    width: int | None = None
    fps: float | None = None
    num_frames: int | None = None
    duration: float | None = None

class VideoReader(abc.ABC):
    def __init__(self, video_source): ...
    def get_metadata(self) -> VideoMetadata: ...
    def extract_frames(self, start_time=0, end_time=None) -> Iterator[np.ndarray]: ...
    def extract_keyframes(self, start_time=0, end_time=None) -> Frames: ...
    def extract_clip(self, start_time=0, end_time=None,
                     output_path=None, to_numpy=True) -> Optional[Clip]: ...
    def close(self) -> None: ...

class AVReader(VideoReader): ...
class FFmpegReader(VideoReader): ...
class DecordReader(VideoReader): ...

def create_video_reader(video_source: str, backend: str = "auto",
                        **kwargs) -> VideoReader: ...

Import

from data_juicer.utils.video_utils import (
    create_video_reader, VideoReader, AVReader, FFmpegReader,
    DecordReader, VideoMetadata, Frames, Clip
)

I/O Contract

Inputs

Name	Type	Required	Description
video_source	Union[str, Path, bytes, IO[bytes]]	Yes	Video file path, URL, raw bytes, or file-like object.
backend	str	No	Backend to use: "auto", "av", "ffmpeg", or "decord". Default "auto".
start_time	float	No	Start time in seconds for frame/clip extraction. Default 0.
end_time	float	No	End time in seconds (exclusive). None means end of video.
output_path	str	No	File path to save extracted clips.
to_numpy	bool	No	Whether to return clip frames as numpy arrays. Default True.
video_stream_index	int	No	Video stream index for multi-stream videos. Default 0.

Outputs

Name	Type	Description
metadata	VideoMetadata	Video properties (height, width, fps, num_frames, duration).
frames	Iterator[np.ndarray]	Iterator of frame arrays from extract_frames.
keyframes	Frames	Container with keyframe arrays, indices, and PTS timestamps.
clip	Clip	Video clip with source, span, optional path/bytes/frames.

Usage Examples

from data_juicer.utils.video_utils import create_video_reader

# Auto-select backend
with create_video_reader("/data/video.mp4") as reader:
    # Get metadata
    meta = reader.metadata
    print(f"Duration: {meta.duration}s, FPS: {meta.fps}")
    print(f"Resolution: {meta.width}x{meta.height}")

    # Extract frames from 5-10 seconds
    for frame in reader.extract_frames(start_time=5.0, end_time=10.0):
        print(f"Frame shape: {frame.shape}")  # (H, W, 3)

    # Extract keyframes
    kf = reader.extract_keyframes()
    print(f"Found {len(kf.frames)} keyframes")
    print(f"Keyframe timestamps: {kf.pts_time}")

    # Extract a clip
    clip = reader.extract_clip(
        start_time=2.0, end_time=8.0,
        output_path="/output/clip.mp4"
    )
    if clip:
        print(f"Clip saved to: {clip.path}")

# Use a specific backend
from data_juicer.utils.video_utils import FFmpegReader
reader = FFmpegReader("/data/video.mp4")
meta = reader.metadata
reader.close()

Related Pages

• Datajuicer_Data_juicer_Multimodal_Utils -- Lower-level video functions (load_video, cut_video_by_seconds) used by AVReader