Implementation:Open compass VLMEvalKit MMHelix Tapa Eval
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Puzzle Solving, Tapa |
Overview
Implements the TapaEvaluator for evaluating Tapa puzzle solutions in the MMHelix benchmark.
Description
The TapaEvaluator extends BaseEvaluator to evaluate Tapa puzzle solutions where black cells must form a single connected group without any 2x2 blocks, and clue numbers indicate lengths of connected black cell groups in the 8 surrounding cells. The extract_answer method extracts coordinates or grid representations from model output, returning typed results (coordinates or grid format). It uses _extract_coordinates and _extract_grid helper methods with markdown code block removal. The evaluate method validates connectivity via BFS, the no-2x2 constraint, and clue satisfaction.
Usage
Called internally by the MMHelix dataset class during Tapa puzzle evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/mmhelix/evaluators/tapa_eval.py, Lines: L1-527 - Import:
from vlmeval.dataset.utils.mmhelix.evaluators.tapa_eval import TapaEvaluator
Key Functions:
class TapaEvaluator(BaseEvaluator):
def prepare_prompt(self, question, params): ...
def extract_answer(self, model_output): ...
def evaluate(self, predicted_answer, ground_truth, params): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Model output string containing black cell coordinates or grid; ground-truth positions; initial state with clue cells |
| Outputs | Boolean indicating whether the black cell placement satisfies all Tapa constraints (connectivity, no-2x2, clue values) |
Usage Examples
# Internal usage example
from vlmeval.dataset.utils.mmhelix.evaluators.tapa_eval import TapaEvaluator
evaluator = TapaEvaluator()
answer = evaluator.extract_answer(model_output)
is_correct = evaluator.evaluate(answer, ground_truth, params)