Implementation:Open compass VLMEvalKit NaturalBench Utils
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, VQA, Compositional Reasoning |
Overview
Provides answer extraction and multi-metric scoring for the NaturalBench visual question-answering benchmark with yes/no and multiple-choice evaluation.
Description
This module implements `extract_answer` which determines answer polarity (yes=1, no=0) from model outputs for both yes_no and multiple_choice task types using word-boundary regex matching with positional priority. The `get_scores` function computes comprehensive metrics from four question-image combinations (q0_i0, q0_i1, q1_i0, q1_i1): individual accuracy, question-level accuracy (both images correct), image-level accuracy, and group-level accuracy (all four correct).
Usage
Called internally by the corresponding dataset class during evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/naturalbench.py, Lines: L1-145 - Import:
from vlmeval.dataset.utils.naturalbench import extract_answer, get_scores
Key Functions:
def extract_answer(output_string, task_type="yes_no"): ...
def get_scores(scores): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Model output strings for answer extraction; score dicts/lists with q0_i0/q0_i1/q1_i0/q1_i1 entries |
| Outputs | Integer answer (1/0/-1); comprehensive score dict with individual, question, image, and group accuracies |
Usage Examples
from vlmeval.dataset.utils.naturalbench import extract_answer, get_scores
answer = extract_answer("Yes, the cat is there", task_type="yes_no")
# answer = 1
Related Pages
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment