Implementation:Open compass VLMEvalKit MME Reasoning
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Reasoning, Multi-modal |
Overview
Provides evaluation utilities for the MME-Reasoning benchmark, supporting structured answer extraction and multi-format scoring with LLM-as-judge capabilities.
Description
This module implements evaluation for MME-Reasoning tasks that require structured answer extraction from model responses. It uses prompt templates (id_answer_pair_prompt, coordinate_answer_prompt) to extract JSON-formatted answers containing ID-answer pairs and coordinate positions via LLM-based parsing. The evaluation supports various answer formats including numeric mappings, letter-ID pairs, and spatial coordinate answers. It includes SymPy-based mathematical expression comparison and regex-based answer extraction for different reasoning task types.
Usage
Called internally by the MME-Reasoning dataset class during evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/mme_reasoning.py, Lines: L1-1200 - Import:
from vlmeval.dataset.utils.mme_reasoning import evaluate_answer
Key Functions:
# Prompt templates for LLM-based answer extraction
id_answer_pair_prompt = "..."
coordinate_answer_prompt = "..."
def extract_answer(response, question, answer_type): ...
def evaluate_answer(prediction, ground_truth, task_type): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Model response text, question text, answer type specification, and ground-truth answers in JSON format |
| Outputs | Extracted JSON answers (ID-value pairs, coordinates); boolean correctness scores per question |
Usage Examples
# Internal usage example
from vlmeval.dataset.utils.mme_reasoning import extract_answer
answer = extract_answer(model_response, question, "id_answer_pair")