Implementation:Open compass VLMEvalKit VGRPBench Evaluation
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Puzzle_Solving |
Overview
Orchestrates atomic evaluation, score parsing, and aggregation for VGRPBench visual grid reasoning puzzles.
Description
This module provides the evaluation pipeline for VGRPBench puzzles, coordinating model output parsing, puzzle solution verification, and score computation. It extracts puzzle type from category metadata, delegates solution checking to the evaluate_single_puzzle function, and aggregates perception and answer correctness scores across puzzle categories. The module also constructs system prompts by loading puzzle-type-specific formatting prompt JSON files.
Usage
Called internally by VGRPBench evaluation pipeline for puzzle constraint verification.
Code Reference
- Source:
vlmeval/dataset/utils/vgrpbench/evaluation.py, Lines: L1-135 - Import:
from vlmeval.dataset.utils.vgrpbench.evaluation import VGRPBench_atomeval, VGRPBench_score, build_prompt
Key Functions:
def get_eval(judge, content): ...
def VGRPBench_atomeval(model, prompt, line): ...
def parse_score(line, output): ...
def VGRPBench_score(data): ...
def build_prompt(line): ...
def VGRPBench_get_system_prompt(line): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Model predictions, puzzle data lines with category, states, and prediction fields |
| Outputs | Dictionary with perception_correct and answer_correct results, or aggregated DataFrame of scores by category |
Usage Examples
# Internal usage
from vlmeval.dataset.utils.vgrpbench.evaluation import VGRPBench_atomeval, VGRPBench_score
scores = VGRPBench_atomeval(model, prompt, line)
result_df = VGRPBench_score(data)
Related Pages
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment