Principle:Open compass VLMEvalKit MCQ Prompt Construction
| Field | Value |
|---|---|
| source | Repo |
| domain | Vision, NLP, Evaluation |
Overview
A prompt formatting pattern that constructs standardized multiple-choice question prompts from benchmark data rows with images, options, and hints.
Description
For MCQ benchmarks, VLMEvalKit constructs a standardized prompt format: optional hint, question text, lettered options (A/B/C/D), and an instruction to select the correct answer. The prompt is paired with one or more images from the data row. This standardized format ensures consistent evaluation across different VLMs. The MCQ prompt builder in ImageMCQDataset.build_prompt() handles: extracting options from columns A-Z, formatting hints, constructing the instruction, and pairing with decoded images.
Usage
Used automatically for all MCQ benchmarks (MMBench, AI2D, MMStar, SEEDBench, etc.). The default behavior can be overridden by VLM adapters via use_custom_prompt()/build_prompt().
Theoretical Basis
Standardized evaluation protocol — consistent prompt formatting ensures fair comparison across models. The format "Hint → Question → Options → Instruction" follows common MCQ conventions.