Implementation:Open compass VLMEvalKit Video MMLU CAP

Field	Value
source	VLMEvalKit
domain	Vision, Benchmarking, Video Academic Understanding

Overview

Benchmark dataset implementation for Video-MMLU academic discipline video understanding evaluation in VLMEvalKit.

Description

Video_MMLU_CAP inherits from VideoBaseDataset and implements the Video MMLU Caption benchmark for evaluating video-based academic understanding. The TYPE field is set to 'Video-VQA'. The file also defines Video_MMLU_QA (also TYPE 'Video-VQA') for question answering. Both support discipline-based subsets (Math, Physics, Chemistry, etc.) with configurable data limits for subsampling.

Usage

Registered in vlmeval/dataset/__init__.py and invoked through build_dataset() by benchmark name.

Code Reference

Source: vlmeval/dataset/video_mmlu.py, Lines: L1-614
Import: from vlmeval.dataset.video_mmlu import Video_MMLU_CAP

Signature:

class Video_MMLU_CAP(VideoBaseDataset):
    TYPE = 'Video-VQA'
    MODALITY = 'VIDEO'
    ...

class Video_MMLU_QA(VideoBaseDataset):
    TYPE = 'Video-VQA'
    ...

I/O Contract

Direction	Description
Inputs	TSV dataset file with academic video paths and discipline questions
Outputs	Evaluation results DataFrame with per-discipline accuracy scores

Usage Examples

from vlmeval.dataset import build_dataset
dataset = build_dataset('Video_MMLU_CAP')

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment