Implementation:Open compass VLMEvalKit Supported Video Datasets
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Video_Understanding, Evaluation |
Overview
Concrete registry of pre-configured video dataset constructors with frame sampling parameters provided by VLMEvalKit.
Description
supported_video_datasets in vlmeval/dataset/video_dataset_config.py (L209-220) aggregates multiple per-benchmark dictionaries. Each entry maps a configuration-specific name to a functools.partial of the dataset class with parameters: dataset (base name), nframe (fixed frame count), fps (frames per second), pack (batch questions per video). The registry includes configurations for MVBench, Video-MME, MMBench-Video, MLVU, LongVideoBench, TempCompass, CGBench, WorldSense, and more. The build_dataset() function checks this registry first before falling back to standard dataset classes.
Usage
Look up video benchmark configurations in this registry. Used automatically by build_dataset().
Code Reference
- Source:
vlmeval/dataset/video_dataset_config.py, Lines: L209-220 (aggregation), L1-208 (individual dicts) - Signature:
# Registry structure
supported_video_datasets: Dict[str, functools.partial] = {}
# Example entries from the config:
mvbench_dataset = {
'MVBench': partial(MVBench, dataset='MVBench', nframe=0, fps=-1),
'MVBench_8frame': partial(MVBench, dataset='MVBench', nframe=8),
'MVBench_16frame': partial(MVBench, dataset='MVBench', nframe=16),
'MVBench_1fps': partial(MVBench, dataset='MVBench', fps=1.0),
}
videomme_dataset = {
'Video-MME': partial(VideoMME, dataset='Video-MME', nframe=0, fps=-1),
'Video-MME_16frame': partial(VideoMME, dataset='Video-MME', nframe=16),
'Video-MME_1fps': partial(VideoMME, dataset='Video-MME', fps=1.0),
}
- Import:
from vlmeval.dataset.video_dataset_config import supported_video_datasets
I/O Contract
| Direction | Name | Type | Description |
|---|---|---|---|
| Input | dataset_name | str | Configuration-specific name (e.g., "MVBench_8frame") |
| Output | (callable) | Callable | Returns VideoBaseDataset subclass instance with configured frame sampling |
Usage Examples
from vlmeval.dataset.video_dataset_config import supported_video_datasets
from vlmeval.dataset import build_dataset
# List available video benchmarks
print(list(supported_video_datasets.keys()))
# Build a specific configuration
dataset = build_dataset("MVBench_8frame")
print(f"Dataset: {dataset.dataset_name}, Frames: {dataset.nframe}")
# Or directly from registry
dataset = supported_video_datasets["Video-MME_1fps"]()