Implementation:Open compass VLMEvalKit MMHelix Minesweeper Eval

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Puzzle Solving, Minesweeper

Overview

Evaluates Minesweeper puzzle solutions in the MMHelix benchmark by comparing predicted mine coordinates with ground truth positions.

Description

The `MinesweeperEvaluator` class extends `BaseEvaluator` to validate Minesweeper solutions by comparing sets of mine coordinates. It supports both string and list input formats for coordinates. The `_extract_coordinates` method parses coordinate strings in formats like "(0,5),(0,7),(1,1)" into sets of (row, col) tuples. Evaluation is a strict set equality check between predicted and ground truth mine positions.

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/mmhelix/evaluators/minesweeper_eval.py, Lines: L1-95
• Import: from vlmeval.dataset.utils.mmhelix.evaluators.minesweeper_eval import MinesweeperEvaluator

Key Functions:

class MinesweeperEvaluator(BaseEvaluator):
    def evaluate(self, predicted_answer, ground_truth, initial_state) -> bool: ...
    def _extract_coordinates(self, coord_str) -> set: ...

I/O Contract

Direction	Description
Inputs	Predicted mine coordinates as string or list; ground truth coordinates as string or list
Outputs	Boolean indicating whether predicted mine positions exactly match ground truth

Usage Examples

from vlmeval.dataset.utils.mmhelix.evaluators.minesweeper_eval import MinesweeperEvaluator

evaluator = MinesweeperEvaluator()
is_correct = evaluator.evaluate("(0,5),(1,2)", "(0,5),(1,2)", None)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction