Implementation:Open compass VLMEvalKit TempCompass Utils
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Video Understanding, Temporal Reasoning |
Overview
Provides evaluation utilities for the TempCompass temporal video understanding benchmark with GPT-based multi-task scoring across caption matching, QA, and captioning.
Description
This module defines system prompts for three evaluation modes: multi-choice QA (`system_prompt_multi_choice`), caption matching (`system_prompt_caption_matching`), and video captioning (`system_prompt_captioning`). The captioning evaluator uses a two-stage pipeline: first generating descriptions from video, then answering multi-choice questions based on descriptions. Each mode outputs "Correct" or "Incorrect" judgments. The module supports format-aware evaluation with reasoning and answer structure.
Usage
Called internally by the corresponding dataset class during evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/tempcompass.py, Lines: L1-254 - Import:
from vlmeval.dataset.utils.tempcompass import system_prompt_multi_choice
Key Functions:
system_prompt_multi_choice = """..."""
system_prompt_caption_matching = """..."""
system_prompt_captioning = """..."""
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Video descriptions, multi-choice questions, ground truth answers, and model predictions |
| Outputs | "Correct" or "Incorrect" judgments from the GPT evaluator |
Usage Examples
from vlmeval.dataset.utils.tempcompass import system_prompt_multi_choice
# Use as system prompt for GPT-based evaluation