Implementation:Open compass VLMEvalKit LogicVista Utils
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Logic, Visual Reasoning |
Overview
Provides answer extraction and evaluation utilities for the LogicVista visual logical reasoning benchmark using GPT-based letter extraction.
Description
This module implements `build_prompt_logicvista` which constructs prompts for a GPT judge to extract multiple-choice letter answers from model responses. The `LogicVista_auxeval` function uses a retry mechanism (up to 5 attempts with increasing temperature) to reliably extract uppercase letter answers. It validates that extracted answers are purely uppercase alphabetic characters and compares them against sorted ground truth answer letters.
Usage
Called internally by the corresponding dataset class during evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/logicvista.py, Lines: L1-150 - Import:
from vlmeval.dataset.utils.logicvista import LogicVista_auxeval, build_prompt_logicvista
Key Functions:
def build_prompt_logicvista(line): ...
def LogicVista_auxeval(model, line): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | A data line dict with 'question', 'prediction', and 'answer' fields; a judge model instance |
| Outputs | Dict with 'log' (evaluation trace) and 'res' (boolean correctness) |
Usage Examples
from vlmeval.dataset.utils.logicvista import LogicVista_auxeval
result = LogicVista_auxeval(judge_model, line)
Related Pages
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment