Implementation:Open compass VLMEvalKit MovieChat1K Utils
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Video Understanding, Movie QA |
Overview
Provides GPT-based scoring and dimension-wise accuracy computation for the MovieChat-1K video question-answering benchmark.
Description
This module defines `CAL_SCORE_PROMPT` for GPT-based evaluation of video QA pairs with yes/no prediction and 0-5 score output. The `get_dimension_rating` function computes per-dimension (global, breakpoint) and overall accuracy/ratings from evaluation results. It includes robust `parse_score_dict` handling for JSON/eval parsing of score dictionaries. Dimensions are organized in `MOVIECHAT1K_DIMENSIONS` with automatic overall aggregation.
Usage
Called internally by the corresponding dataset class during evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/moviechat1k.py, Lines: L1-111 - Import:
from vlmeval.dataset.utils.moviechat1k import get_dimension_rating
Key Functions:
def get_dimension_rating(data_path): ...
CAL_SCORE_PROMPT = """..."""
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Path to evaluation results file with per-sample scores and mode labels |
| Outputs | Dictionary with per-dimension accuracy and rating scores |
Usage Examples
from vlmeval.dataset.utils.moviechat1k import get_dimension_rating
ratings = get_dimension_rating("results.xlsx")
Related Pages
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment