Implementation:Open compass VLMEvalKit ShortQA Utils
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, VQA, Short Answer, LLM Judge |
Overview
Provides LLM-judge-based evaluation for short-form visual question-answering tasks with bilingual (English/Chinese) support and specialized evaluation templates.
Description
This module implements `EVAL_TMPL` and `EVAL_TMPL_CN` evaluation templates that instruct a GPT judge to determine semantic correctness of model responses compared to ground truth. The evaluation considers phrasing variations and additional relevant details as correct, while contradictions and missing essential information are marked incorrect. It supports configurable requirements and example-based guidance through template placeholders for `{requirement}`, `{examples}`, `{question}`, `{ground_truth}`, and `{response}`.
Usage
Called internally by the corresponding dataset class during evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/shortqa.py, Lines: L1-276 - Import:
from vlmeval.dataset.utils.shortqa import EVAL_TMPL, EVAL_TMPL_CN
Key Functions:
EVAL_TMPL = """...""" # English evaluation template
EVAL_TMPL_CN = """...""" # Chinese evaluation template
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Question, model response, ground truth answer, optional requirements and examples |
| Outputs | Correctness ("yes"/"no") and reason string from the GPT judge |
Usage Examples
from vlmeval.dataset.utils.shortqa import EVAL_TMPL
prompt = EVAL_TMPL.format(
requirement="", examples="",
question="What color is the car?",
ground_truth="Red",
response="The car is red."
)