Implementation:Open compass VLMEvalKit ImageShortQADataset

Field	Value
source	VLMEvalKit
domain	Vision, Benchmarking, Short Answer QA

Overview

Benchmark dataset implementation for Short QA evaluation in VLMEvalKit.

Description

ImageShortQADataset inherits from ImageBaseDataset and implements short-answer visual QA evaluation. The TYPE field is set to 'Short'. It supports LiveMMBench splits and HLE dataset, with auxiliary evaluation using judge models. The file also defines PathVQA_VAL and PathVQA_TEST subclasses.

Usage

Registered in vlmeval/dataset/__init__.py and invoked through build_dataset() by benchmark name.

Code Reference

Source: vlmeval/dataset/image_shortqa.py, Lines: L1-160
Import: from vlmeval.dataset.image_shortqa import ImageShortQADataset

Signature:

class ImageShortQADataset(ImageBaseDataset):
    TYPE = 'Short'
    DATASET_URL = {...}
    DATASET_MD5 = {...}
    ...

I/O Contract

Direction	Description
Inputs	TSV dataset file with image/video paths and questions
Outputs	Evaluation results DataFrame with scores per category

Usage Examples

from vlmeval.dataset import build_dataset
dataset = build_dataset('hle')

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment