Implementation:Open compass VLMEvalKit MMDU Utils

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Multi-modal Dialogue, Creative Assessment

Overview

Provides GPT-based multi-dimensional evaluation for the MMDU (Multi-Modal Dialogue Understanding) benchmark with creative text assessment criteria.

Description

This module defines a comprehensive `meta_prompt` with detailed scoring rubrics across six dimensions: Creativity (1-10), Richness (1-10), Visual Perception (1-10), Logical Coherence (1-10), Answer Accuracy (1-10), and Image Relationship Understanding (1-10). Each dimension has explicit scoring bands with descriptions. The evaluation compares AI model responses against reference answers, producing both per-dimension and overall scores with rigorous scoring guidelines.

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/mmdu.py, Lines: L1-126
• Import: from vlmeval.dataset.utils.mmdu import meta_prompt

Key Functions:

meta_prompt = """..."""  # Multi-dimensional scoring rubric

I/O Contract

Direction	Description
Inputs	Model response, reference answer, and creative question for evaluation
Outputs	Per-dimension scores (1-10) and overall score (1-10)

Usage Examples

from vlmeval.dataset.utils.mmdu import meta_prompt

# Use meta_prompt as system prompt for GPT judge

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction