Implementation:Open compass VLMEvalKit Video MMLU Utils

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Video Understanding, Academic Knowledge

Overview

Provides GPT-based evaluation utilities for the Video-MMLU benchmark with strict OCR and reasoning-specific scoring criteria.

Description

This module defines two distinct scoring system prompts: `SYSTEM_CAL_SCORE_PROMPT_CAP` for OCR-related questions (requiring strict letter-by-letter comparison) and `SYSTEM_CAL_SCORE_PROMPT_QA` for reasoning-based questions (requiring critical concept preservation). It also includes a description-based QA generation pipeline. Dimensions cover academic subjects (math, physics, chemistry, biology) and are organized in `VIDEO_MMLU_DIMENSIONS` for per-subject and overall accuracy computation.

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/video_mmlu.py, Lines: L1-161
• Import: from vlmeval.dataset.utils.video_mmlu import VIDEO_MMLU_DIMENSIONS

Key Functions:

SYSTEM_CAL_SCORE_PROMPT_CAP = """..."""
SYSTEM_CAL_SCORE_PROMPT_QA = """..."""
VIDEO_MMLU_DIMENSIONS = {...}

I/O Contract

Direction	Description
Inputs	Video descriptions, academic questions, correct answers, and predicted answers
Outputs	Dict string with 'pred' (yes/no) and 'score' (0-5) keys

Usage Examples

from vlmeval.dataset.utils.video_mmlu import VIDEO_MMLU_DIMENSIONS

print(VIDEO_MMLU_DIMENSIONS.keys())

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction