Implementation:Open compass VLMEvalKit ImageBaseDataset

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Data_Processing

Overview

Concrete base class for all image-based benchmark datasets in VLMEvalKit providing data loading, prompt construction, and image handling.

Description

ImageBaseDataset in vlmeval/dataset/image_base.py (L32-207) is the root class for image benchmarks. Constructor takes dataset name and skip_noimg flag.

Key methods:

• load_data(dataset) — fetches TSV from URL if not cached, verifies MD5, and returns DataFrame
• prepare_tsv(url, file_md5) — handles download, caching, and localization for large files
• dump_image(line) — decodes base64 images to disk files
• build_prompt(line) — constructs default message list with images and question text
• supported_datasets() — classmethod that returns list of supported dataset names from DATASET_URL keys

Subclasses ImageMCQDataset and ImageVQADataset add type-specific evaluation and prompt formatting.

Usage

Subclass for new image benchmarks. Set DATASET_URL and DATASET_MD5 class attributes.

Code Reference

• Source: vlmeval/dataset/image_base.py, Lines: L32-207
• Also: vlmeval/dataset/image_mcq.py, Lines: L42-465 (ImageMCQDataset); vlmeval/dataset/image_vqa.py, Lines: L16-123 (ImageVQADataset)

Signature:

class ImageBaseDataset:
    MODALITY = 'IMAGE'
    DATASET_URL = {}   # Override with {name: url} mapping
    DATASET_MD5 = {}   # Override with {name: md5} mapping

    def __init__(self, dataset: str = 'MMBench', skip_noimg: bool = True): ...
    def load_data(self, dataset: str) -> pd.DataFrame: ...
    def prepare_tsv(self, url: str, file_md5: Optional[str] = None) -> pd.DataFrame: ...
    def dump_image(self, line: pd.Series) -> List[str]: ...
    def build_prompt(self, line: Union[int, pd.Series]) -> List[Dict]: ...
    @classmethod
    def supported_datasets(cls) -> List[str]: ...
    @abstractmethod
    def evaluate(self, eval_file: str, **judge_kwargs): ...

Import:

from vlmeval.dataset.image_base import ImageBaseDataset

I/O Contract

Direction	Name	Type	Description
Input	dataset	str	Benchmark name
Input	skip_noimg	bool	Skip rows without images
Output	self.data	DataFrame	Loaded benchmark data
Output	self.dataset_name	str	Resolved dataset name
Output	self.img_root	str	Path to extracted images directory
Output	build_prompt return	List[Dict]	Message list with type/value keys (images + question text)

Usage Examples

from vlmeval.dataset.image_mcq import ImageMCQDataset

class MyBenchmark(ImageMCQDataset):
    TYPE = 'MCQ'
    DATASET_URL = {
        'MyBench_v1': 'https://example.com/mybench_v1.tsv',
    }
    DATASET_MD5 = {
        'MyBench_v1': 'abc123def456...',
    }

# Now build_dataset("MyBench_v1") will return an instance

Related Pages

• Principle:Open_compass_VLMEvalKit_Dataset_Base_Class_Hierarchy