Implementation:Open compass VLMEvalKit OSTDataset

Field	Value
source	VLMEvalKit
domain	Vision, Benchmarking, Interleaved Learning

Overview

Benchmark dataset implementation for OST (One-Shot Transfer) interleaved learning evaluation in VLMEvalKit.

Description

OSTDataset inherits from ImageILDataset (which itself inherits from ImageBaseDataset) and implements the OST benchmark for evaluating interleaved image-text learning capabilities. The parent class ImageILDataset has TYPE set to 'IL' and handles multi-image prompts with ImageHere placeholders. OSTDataset adds custom evaluation logic with per-category scoring tracked through Excel-based result files.

Usage

Registered in vlmeval/dataset/__init__.py and invoked through build_dataset() by benchmark name.

Code Reference

Source: vlmeval/dataset/ost_bench.py, Lines: L1-306
Import: from vlmeval.dataset.ost_bench import OSTDataset

Signature:

class ImageILDataset(ImageBaseDataset):
    TYPE = 'IL'
    DATASET_URL = {...}
    DATASET_MD5 = {...}
    ...

class OSTDataset(ImageILDataset):
    ...

I/O Contract

Direction	Description
Inputs	TSV dataset file with interleaved image-text content and questions
Outputs	Evaluation results DataFrame with per-category accuracy scores

Usage Examples

from vlmeval.dataset import build_dataset
dataset = build_dataset('OST')

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment