Implementation:Open compass VLMEvalKit MVBench

Field	Value
source	VLMEvalKit
domain	Vision, Benchmarking, Video Understanding, Multiple Choice

Overview

Benchmark dataset implementation for MVBench multi-task video understanding evaluation in VLMEvalKit.

Description

MVBench inherits from VideoBaseDataset and implements the MVBench benchmark for evaluating video understanding across 20 temporal reasoning tasks. The TYPE field is set to 'Video-MCQ'. Tasks include Action Sequence, Action Prediction, Fine-grained Action, Object Existence, Scene Transition, and more. The file also defines MVBench_MP4, a variant that works with MP4 video files directly rather than individual task JSON files.

Usage

Registered in vlmeval/dataset/__init__.py and invoked through build_dataset() by benchmark name.

Code Reference

Source: vlmeval/dataset/mvbench.py, Lines: L1-669
Import: from vlmeval.dataset.mvbench import MVBench

Signature:

class MVBench(VideoBaseDataset):
    TYPE = 'Video-MCQ'
    MD5 = 'fd21d36522cdedd46d84dc46715ad832'
    DEFAULT_JUDGE = ['chatgpt-0125', 'gpt-4-0125']
    ...

class MVBench_MP4(VideoBaseDataset):
    TYPE = 'Video-MCQ'
    ...

I/O Contract

Direction	Description
Inputs	TSV dataset file with video paths and temporal reasoning MCQ questions
Outputs	Evaluation results DataFrame with accuracy per task type

Usage Examples

from vlmeval.dataset import build_dataset
dataset = build_dataset('MVBench')

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment