Implementation:Open compass VLMEvalKit TempCompass

Field	Value
source	VLMEvalKit
domain	Vision, Benchmarking, Temporal Video Understanding

Overview

Benchmark dataset implementation for TempCompass temporal video understanding evaluation in VLMEvalKit.

Description

TempCompass inherits from ConcatVideoDataset and aggregates three sub-datasets: TempCompass_MCQ (TYPE 'Video-MCQ'), TempCompass_Captioning (TYPE 'Video-VQA'), and TempCompass_YorN (TYPE 'Video-Y/N'). It provides comprehensive temporal understanding evaluation across multiple dimensions and task types, computing per-dimension and per-task-type average scores alongside overall accuracy.

Usage

Registered in vlmeval/dataset/__init__.py and invoked through build_dataset() by benchmark name.

Code Reference

Source: vlmeval/dataset/tempcompass.py, Lines: L1-646
Import: from vlmeval.dataset.tempcompass import TempCompass

Signature:

class TempCompass(ConcatVideoDataset):
    ...

class TempCompass_MCQ(VideoBaseDataset):
    TYPE = 'Video-MCQ'
    ...

class TempCompass_Captioning(VideoBaseDataset):
    TYPE = 'Video-VQA'
    ...

class TempCompass_YorN(VideoBaseDataset):
    TYPE = 'Video-Y/N'
    ...

I/O Contract

Direction	Description
Inputs	TSV dataset file with video paths and temporal understanding questions
Outputs	Evaluation results DataFrame with per-dimension and per-task-type scores

Usage Examples

from vlmeval.dataset import build_dataset
dataset = build_dataset('TempCompass')

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment