Implementation:Open compass VLMEvalKit MMIFEval

Field	Value
source	VLMEvalKit
domain	Vision, Benchmarking, Instruction Following

Overview

Benchmark dataset implementation for MM-IFEval multimodal instruction following evaluation in VLMEvalKit.

Description

MMIFEval inherits from ImageBaseDataset and implements the MM-IFEval benchmark for evaluating how well multimodal models follow complex instructions. The TYPE field is set to 'VQA'. It includes a multi-level evaluation system (A-Level, B-Level, C-Level) for scoring instruction adherence and provides comprehensive judge-based evaluation functionality.

Usage

Registered in vlmeval/dataset/__init__.py and invoked through build_dataset() by benchmark name.

Code Reference

Source: vlmeval/dataset/mmifeval.py, Lines: L1-483
Import: from vlmeval.dataset.mmifeval import MMIFEval

Signature:

class MMIFEval(ImageBaseDataset):
    TYPE = "VQA"
    DATASET_URL = {...}
    DATASET_MD5 = {...}
    ...

I/O Contract

Direction	Description
Inputs	TSV dataset file with images and complex instruction-following tasks
Outputs	Evaluation results DataFrame with multi-level instruction adherence scores

Usage Examples

from vlmeval.dataset import build_dataset
dataset = build_dataset('MM-IFEval')

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment