Implementation:Open compass VLMEvalKit MLVU Utils
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Video Understanding, Multi-task |
Overview
Provides evaluation utilities for the MLVU (Multi-task Long Video Understanding) benchmark with GPT-based scoring for multiple video understanding tasks.
Description
This module defines system prompts for two evaluation modes: sub-scene evaluation (accuracy + relevance scoring on 1-5 scales) and video summarization evaluation (completeness + reliability scoring on 1-5 scales). The `check_ans_with_model` function uses a GPT judge to assess model answers against ground truth, outputting JSON-formatted scores. It supports both MCQ (multiple-choice question) and open-ended generation evaluation with structured scoring criteria.
Usage
Called internally by the corresponding dataset class during evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/mlvu.py, Lines: L1-189 - Import:
from vlmeval.dataset.utils.mlvu import check_ans_with_model
Key Functions:
def check_ans_with_model(pred, gt, model, item, dataset_name='MLVU_MCQ'): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Prediction string, ground truth string, judge model instance, data item dict, and dataset name |
| Outputs | Boolean correctness flag or JSON score dict with accuracy/relevance or completeness/reliability scores |
Usage Examples
from vlmeval.dataset.utils.mlvu import check_ans_with_model
result = check_ans_with_model(pred, gt, judge_model, item)
Related Pages
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment