Implementation:Open compass VLMEvalKit MMHelix Bridges Eval

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Puzzle Solving, Bridges

Overview

Implements the BridgesEvaluator for evaluating Bridges (Hashiwokakero) puzzle solutions in the MMHelix benchmark.

Description

The BridgesEvaluator extends BaseEvaluator to handle bridge specification extraction and validation. The extract_answer method uses multiple regex patterns to handle diverse answer formats including XML-style tags, JSON-like structures, direct text, Chinese answer formats, and code blocks. It extracts bridge specifications describing connections between island nodes. The evaluate method validates that the bridge network satisfies all island degree constraints and forms a connected graph.

Usage

Called internally by the MMHelix dataset class during Bridges puzzle evaluation.

Code Reference

• Source: vlmeval/dataset/utils/mmhelix/evaluators/bridges_eval.py, Lines: L1-391
• Import: from vlmeval.dataset.utils.mmhelix.evaluators.bridges_eval import BridgesEvaluator

Key Functions:

class BridgesEvaluator(BaseEvaluator):
    def prepare_prompt(self, question, params): ...
    def extract_answer(self, model_output): ...
    def evaluate(self, predicted_answer, ground_truth, initial_state): ...

I/O Contract

Direction	Description
Inputs	Model output string containing bridge specifications; ground-truth bridge set; initial island layout
Outputs	Boolean indicating whether the predicted Bridges solution forms a valid connected network

Usage Examples

# Internal usage example
from vlmeval.dataset.utils.mmhelix.evaluators.bridges_eval import BridgesEvaluator
evaluator = BridgesEvaluator()
bridges = evaluator.extract_answer(model_output)
is_correct = evaluator.evaluate(bridges, ground_truth, initial_state)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction