Workflow:NVIDIA NeMo Curator Video Curation Pipeline
| Knowledge Sources | |
|---|---|
| Domains | Data_Engineering, Video_Processing, Generative_AI |
| Last Updated | 2026-02-14 17:00 GMT |
Overview
End-to-end process for curating high-quality video clip datasets from raw video files for training video generation models and world foundation models using NeMo Curator's GPU-accelerated pipeline.
Description
This workflow outlines the standard procedure for processing raw long-form videos into curated, high-quality video clips suitable for training generative video models, video world models, and video retrieval systems. It covers video ingestion, scene-based or fixed-stride clipping, clip transcoding, frame extraction, quality and motion filtering, caption generation, embedding computation, semantic deduplication, and final export. The pipeline uses the XennaExecutor backend by default and leverages NVIDIA hardware decoders (NVDEC) and encoders (NVENC) for GPU-accelerated video processing. The pipeline is designed to handle 100+ PB of video data across distributed GPU clusters.
Usage
Execute this workflow when you have a collection of raw video files (local or cloud-stored) and need to produce a curated dataset of short video clips with associated metadata (captions, embeddings, quality scores) for downstream model training or retrieval applications.
Execution Steps
Step 1: Video Ingestion
Load raw video files into the pipeline from local storage or S3-compatible cloud storage. The VideoReaderStage discovers and reads video files, creating task objects that carry video metadata (duration, resolution, codec information). The FilePartitioningStage groups input file paths into partitions for parallel processing. Videos can be provided as directory paths, file lists, or JSON manifests pointing to remote URLs.
Key considerations:
- Supports local filesystem paths and S3-compatible cloud storage
- JSON file lists enable explicit control over which videos to process
- File partitioning balances work across available GPU workers
- Video metadata (duration, resolution, frame rate) is extracted during ingestion
Step 2: Video Clipping
Split long videos into shorter clips using either scene-change detection or fixed-stride extraction. The TransNetV2ClipExtractionStage uses a deep neural network (TransNetV2) to detect shot boundaries and extract scene-based clips. The FixedStrideExtractorStage produces clips of uniform duration at regular intervals. The ClipTranscodingStage then encodes the extracted clips to H.264 using either GPU (NVENC) or CPU (libopenh264/libx264) encoders.
Key considerations:
- TransNetV2 detects scene transitions using a trained shot boundary detection model
- Fixed-stride extraction produces clips of configurable length (e.g., 10 seconds) with optional overlap
- Clip transcoding supports GPU-accelerated H.264 encoding via NVENC
- Minimum and maximum clip duration thresholds filter out very short or very long clips
Step 3: Frame Extraction
Extract representative frames from video clips for downstream analysis stages. The VideoFrameExtractionStage extracts frames from full videos, while ClipFrameExtractionStage extracts frames from individual clips. Frames are decoded using NVDEC for GPU-accelerated processing or FFmpeg for CPU-based decoding. Extracted frames are used by subsequent filtering, captioning, and embedding stages.
Key considerations:
- NVDEC provides hardware-accelerated frame decoding on NVIDIA GPUs
- Configurable frame extraction rate (e.g., 1 frame per second, or N frames per clip)
- Frame resolution and format can be configured for downstream model requirements
- Extracted frames are stored as part of the task metadata for subsequent stages
Step 4: Clip Filtering
Apply quality filters to remove low-quality or undesirable clips. The MotionFilterStage analyzes motion vectors to detect and remove static or nearly-static clips. The ClipAestheticFilterStage uses a CLIP-based aesthetic quality scorer to filter clips based on visual appeal of their representative frames. Each filter computes a score and applies a configurable threshold to decide which clips to retain.
Key considerations:
- Motion filtering uses motion vector analysis from video codec metadata
- Aesthetic filtering applies the CLIP ViT-L/14 model plus a trained aesthetic MLP
- Thresholds for each filter are configurable to balance quality vs. dataset size
- Filters can be applied independently or chained in sequence
Step 5: Caption Generation
Generate natural language captions for video clips using vision-language models. The CaptionPreparationStage extracts and formats frames for the captioning model. The CaptionGenerationStage uses the Qwen2.5-VL-7B-Instruct model to generate initial captions from selected frames. The CaptionEnhancementStage optionally refines captions using the Qwen2.5-14B-Instruct text-only model for improved detail and coherence.
Key considerations:
- Caption preparation selects representative frames and formats them for the vision-language model
- Qwen VL model processes video frames with configurable prompts
- Caption enhancement uses a text-only LLM to rewrite and improve initial captions
- Captions are stored as clip-level metadata for training and retrieval
Step 6: Embedding Computation
Generate dense vector embeddings for each video clip using the CosmosEmbed1 multimodal embedding model. The embedding stages process clip frames through the Cosmos-Embed1 model to produce fixed-dimensional vectors that capture the semantic content of each clip. These embeddings enable semantic search, clustering, and deduplication.
Key considerations:
- Cosmos-Embed1 produces multimodal embeddings from video frames
- Embeddings are stored as Parquet files with clip IDs for efficient lookup
- GPU memory requirements scale with batch size and model complexity
- Embeddings are used by the subsequent deduplication stage
Step 7: Semantic Deduplication
Remove near-duplicate video clips using embedding-based semantic deduplication. This reuses the SemanticDeduplicationWorkflow which performs KMeans clustering on clip embeddings, computes pairwise cosine similarity within clusters, and identifies duplicates above a configurable similarity threshold. The deduplication process is especially important for video datasets where similar clips may appear across different source videos.
Key considerations:
- KMeans clustering groups similar clips; number of clusters affects granularity
- Pairwise similarity within clusters identifies near-duplicates
- Epsilon threshold controls the similarity cutoff for duplicate identification
- Deduplication significantly reduces dataset size while preserving diversity
Step 8: Export
Write curated video clips and their associated metadata to the output format. The ClipWriterStage writes transcoded clip files to the output directory along with Parquet metadata files containing captions, embeddings, quality scores, and other computed attributes. Output can target local filesystem or S3-compatible cloud storage.
Key considerations:
- Clip files are written as individual video files (H.264/MP4)
- Metadata is exported as Parquet files with configurable schemas
- Embedding Parquet files are produced separately for retrieval and deduplication
- WebP preview images can optionally be generated via the PreviewStage
- Output directory structure supports WebDataset-style packaging for training