Implementation:Open compass VLMEvalKit CMMMU

Field	Value
source	VLMEvalKit
domain	Vision, Benchmarking, Chinese Multimodal Understanding

Overview

Benchmark dataset implementation for CMMMU (Chinese Massive Multi-discipline Multimodal Understanding) evaluation in VLMEvalKit.

Description

CMMMU inherits from ImageBaseDataset and implements the CMMMU benchmark for evaluating multimodal models on Chinese-language multi-discipline understanding tasks. The TYPE field is set to 'VQA'. It supports single and multi-image question formats with custom image dumping logic for handling multiple images per question.

Usage

Registered in vlmeval/dataset/__init__.py and invoked through build_dataset() by benchmark name.

Code Reference

Source: vlmeval/dataset/cmmmu.py, Lines: L1-354
Import: from vlmeval.dataset.cmmmu import CMMMU

Signature:

class CMMMU(ImageBaseDataset):
    TYPE = 'VQA'
    DATASET_URL = {...}
    DATASET_MD5 = {...}
    ...

I/O Contract

Direction	Description
Inputs	TSV dataset file with images and multi-discipline questions in Chinese
Outputs	Evaluation results DataFrame with scores per category

Usage Examples

from vlmeval.dataset import build_dataset
dataset = build_dataset('CMMMU_VAL')

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment