Implementation:Open compass VLMEvalKit MovieChat1K Utils

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Video Understanding, Movie QA

Overview

Provides GPT-based scoring and dimension-wise accuracy computation for the MovieChat-1K video question-answering benchmark.

Description

This module defines `CAL_SCORE_PROMPT` for GPT-based evaluation of video QA pairs with yes/no prediction and 0-5 score output. The `get_dimension_rating` function computes per-dimension (global, breakpoint) and overall accuracy/ratings from evaluation results. It includes robust `parse_score_dict` handling for JSON/eval parsing of score dictionaries. Dimensions are organized in `MOVIECHAT1K_DIMENSIONS` with automatic overall aggregation.

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/moviechat1k.py, Lines: L1-111
• Import: from vlmeval.dataset.utils.moviechat1k import get_dimension_rating

Key Functions:

def get_dimension_rating(data_path): ...
CAL_SCORE_PROMPT = """..."""

I/O Contract

Direction	Description
Inputs	Path to evaluation results file with per-sample scores and mode labels
Outputs	Dictionary with per-dimension accuracy and rating scores

Usage Examples

from vlmeval.dataset.utils.moviechat1k import get_dimension_rating

ratings = get_dimension_rating("results.xlsx")

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction