Implementation:Open compass VLMEvalKit MMHelix Kukurasu Eval

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Puzzle Solving, Kukurasu

Overview

Evaluates Kukurasu puzzle solutions in the MMHelix benchmark by verifying row and column constraint satisfaction for black/white grid filling.

Description

The `KukurasuEvaluator` class extends `BaseEvaluator` to validate Kukurasu puzzle solutions where black cells (1) and white cells (0) must satisfy row and column sum constraints. Each row's sum of black cell column positions must equal the given row constraint, and each column's sum of black cell row positions must equal the given column constraint. The `extract_answer` method parses 2D binary arrays from model output using regex and JSON parsing.

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/mmhelix/evaluators/kukurasu_eval.py, Lines: L1-156
• Import: from vlmeval.dataset.utils.mmhelix.evaluators.kukurasu_eval import KukurasuEvaluator

Key Functions:

class KukurasuEvaluator(BaseEvaluator):
    def extract_answer(self, model_output) -> List[List[int]]: ...
    def evaluate(self, predicted_answer, ground_truth, params) -> bool: ...

I/O Contract

Direction	Description
Inputs	Model output string with a binary 2D grid; puzzle params with row and column constraints
Outputs	Boolean indicating whether the grid satisfies all Kukurasu constraints

Usage Examples

from vlmeval.dataset.utils.mmhelix.evaluators.kukurasu_eval import KukurasuEvaluator

evaluator = KukurasuEvaluator()
grid = evaluator.extract_answer(model_output)
is_correct = evaluator.evaluate(grid, ground_truth, params)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction