Implementation:Open compass VLMEvalKit VideoMME Utils
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Video Understanding, Multi-category |
Overview
Provides answer extraction and multi-dimensional accuracy computation for the Video-MME (Video Multi-Modal Evaluation) benchmark.
Description
This module defines comprehensive categorization structures for Video-MME evaluation: `DURATIONS` (short/medium/long), `DOMAINS` (Knowledge, Film & Television, Sports, etc.), `SUB_CATEGORIES` (50+ fine-grained categories), and `TASK_CATEGORIES` (Temporal Perception, Spatial Perception, etc.). It uses `extract_answer_from_item` for multiple-choice answer extraction and organizes results across these hierarchical dimensions for detailed performance breakdown.
Usage
Called internally by the corresponding dataset class during evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/videomme.py, Lines: L1-150 - Import:
from vlmeval.dataset.utils.videomme import DURATIONS, DOMAINS, SUB_CATEGORIES
Key Functions:
DURATIONS = ['short', 'medium', 'long']
DOMAINS = ['Knowledge', 'Film & Television', ...]
SUB_CATEGORIES = ['Humanity & History', ...]
TASK_CATEGORIES = ['Temporal Perception', ...]
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Model predictions and ground truth answers with duration, domain, and category metadata |
| Outputs | Per-dimension accuracy breakdowns across durations, domains, sub-categories, and task categories |
Usage Examples
from vlmeval.dataset.utils.videomme import DOMAINS, DURATIONS
print(DOMAINS)
Related Pages
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment