Implementation:Open compass VLMEvalKit VLM2Bench Utils
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Video Understanding, Multi-task |
Overview
Provides multi-task evaluation utilities for VLM2Bench covering true/false pair evaluation, counting tasks, and group (MCQ) tasks.
Description
This module implements evaluation for multiple sub-task types: `parse_tf_answer` extracts True/False answers from model output supporting multiple formats (T/F, True/False), `common_doc_to_text` and `common_doc_to_target` extract questions and answers from samples. The TF pair task evaluation handles positive/negative sample pairs (distinguished by _p/_n suffix) where both must be correct for pair-level accuracy. Counting tasks use `parse_cnt_answer` for numerical extraction, and group tasks use `parse_grp_answer` for multiple-choice letter extraction.
Usage
Called internally by the corresponding dataset class during evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/vlm2bench.py, Lines: L1-243 - Import:
from vlmeval.dataset.utils.vlm2bench import parse_tf_answer, common_process_results
Key Functions:
def parse_tf_answer(model_answer): ...
def common_doc_to_text(sample, **kwargs): ...
def common_process_results(results): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Model answer strings for T/F, counting, or MCQ tasks; sample dicts with index, question, answer, category, prediction |
| Outputs | Parsed answers ('T'/'F', int counts, or letter options); per-category accuracy scores |
Usage Examples
from vlmeval.dataset.utils.vlm2bench import parse_tf_answer
answer = parse_tf_answer("The statement is True.")