Implementation:Open compass VLMEvalKit MLVU Utils

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Video Understanding, Multi-task

Overview

Provides evaluation utilities for the MLVU (Multi-task Long Video Understanding) benchmark with GPT-based scoring for multiple video understanding tasks.

Description

This module defines system prompts for two evaluation modes: sub-scene evaluation (accuracy + relevance scoring on 1-5 scales) and video summarization evaluation (completeness + reliability scoring on 1-5 scales). The `check_ans_with_model` function uses a GPT judge to assess model answers against ground truth, outputting JSON-formatted scores. It supports both MCQ (multiple-choice question) and open-ended generation evaluation with structured scoring criteria.

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/mlvu.py, Lines: L1-189
• Import: from vlmeval.dataset.utils.mlvu import check_ans_with_model

Key Functions:

def check_ans_with_model(pred, gt, model, item, dataset_name='MLVU_MCQ'): ...

I/O Contract

Direction	Description
Inputs	Prediction string, ground truth string, judge model instance, data item dict, and dataset name
Outputs	Boolean correctness flag or JSON score dict with accuracy/relevance or completeness/reliability scores

Usage Examples

from vlmeval.dataset.utils.mlvu import check_ans_with_model

result = check_ans_with_model(pred, gt, judge_model, item)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction