Implementation:Open compass VLMEvalKit MMHelix Aquarium Eval

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Puzzle Solving, Aquarium

Overview

Implements the AquariumEvaluator for evaluating Aquarium puzzle solutions in the MMHelix benchmark.

Description

The AquariumEvaluator extends BaseEvaluator to handle Aquarium puzzle evaluation, where models must identify coordinates of water-filled cells. The extract_answer method employs multiple extraction strategies with decreasing priority: answer blocks, standard list formats, scattered coordinate pairs, and CSV formats. It normalizes text, handles empty list responses, and validates coordinate formats. The evaluate method checks whether predicted coordinates match the ground truth, considering the puzzle's constraint rules.

Usage

Called internally by the MMHelix dataset class during Aquarium puzzle evaluation.

Code Reference

• Source: vlmeval/dataset/utils/mmhelix/evaluators/aquarium_eval.py, Lines: L1-526
• Import: from vlmeval.dataset.utils.mmhelix.evaluators.aquarium_eval import AquariumEvaluator

Key Functions:

class AquariumEvaluator(BaseEvaluator):
    def prepare_prompt(self, question, params): ...
    def extract_answer(self, model_output): ...
    def evaluate(self, predicted_answer, ground_truth, initial_state, params=None): ...

I/O Contract

Direction	Description
Inputs	Model output string containing coordinate lists; ground-truth coordinate set; initial puzzle state
Outputs	Boolean indicating whether the predicted Aquarium solution is correct

Usage Examples

# Internal usage example
from vlmeval.dataset.utils.mmhelix.evaluators.aquarium_eval import AquariumEvaluator
evaluator = AquariumEvaluator()
answer = evaluator.extract_answer(model_output)
is_correct = evaluator.evaluate(answer, ground_truth, initial_state)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction