Principle:Open compass VLMEvalKit Dataset Registration

Field	Value
source	Repo
domain	Vision, Evaluation, Software_Design

Overview

A registration pattern for adding new benchmark datasets to VLMEvalKit's dataset discovery system.

Description

After implementing a dataset class, it must be registered so that build_dataset() can discover it. Registration involves:

1. Implement the dataset class with DATASET_URL and DATASET_MD5 class attributes
2. Import the class in vlmeval/dataset/__init__.py
3. Add the class to the appropriate list (IMAGE_DATASET, VIDEO_DATASET, or CUSTOM_DATASET)

The framework then auto-discovers supported dataset names via each class's supported_datasets() classmethod and populates the SUPPORTED_DATASETS list. For video datasets, an additional step registers partial() entries in supported_video_datasets in vlmeval/dataset/video_dataset_config.py.

Usage

Follow this pattern when adding any new benchmark to VLMEvalKit.

Theoretical Basis

Auto-discovery registration — classes declare their supported names, the framework aggregates them into a central lookup. Similar to plugin registration.

Related Pages

• Implementation:Open_compass_VLMEvalKit_Register_Dataset