Implementation:Open compass VLMEvalKit Verifier
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Answer Verification, LLM-as-Judge |
Overview
Provides LLM-as-judge answer verification prompts for general-purpose answer correctness evaluation across VLMEvalKit benchmarks.
Description
This module implements answer verification prompts including QUESTION_QUALITY_PROMPT_EN_NO_COT for direct judgement and QUESTION_QUALITY_PROMPT_EN_COT for chain-of-thought evaluation. Both prompts define comprehensive evaluation criteria: referencing standard answers as definitive, comparing only final answers (ignoring reasoning errors), handling mathematical expression equivalence, supporting multiple-answer questions, processing boxed LaTeX answers, and detecting invalid responses. The evaluation outputs are graded as A (CORRECT), B (INCORRECT), or C (INVALID), with the CoT variant providing structured evaluation protocol with step-by-step comparison methodology.
Usage
Called internally by various dataset classes when LLM-based answer verification is needed.
Code Reference
- Source:
vlmeval/dataset/utils/verifier.py, Lines: L1-353 - Import:
from vlmeval.dataset.utils.verifier import QUESTION_QUALITY_PROMPT_EN_NO_COT, QUESTION_QUALITY_PROMPT_EN_COT
Key Functions:
QUESTION_QUALITY_PROMPT_EN_NO_COT = "..." # Direct evaluation prompt (A/B/C grading)
QUESTION_QUALITY_PROMPT_EN_COT = "..." # Chain-of-thought evaluation prompt
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Original question, standard answer, and candidate's answer strings formatted into the prompt template |
| Outputs | Single letter grade: A (CORRECT), B (INCORRECT), or C (INVALID) |
Usage Examples
# Internal usage example
from vlmeval.dataset.utils.verifier import QUESTION_QUALITY_PROMPT_EN_NO_COT
prompt = QUESTION_QUALITY_PROMPT_EN_NO_COT.format(
question="What is 2+2?",
gold_answer="4",
llm_response="The answer is 4."
)