Implementation:Open compass VLMEvalKit MMHelix Kakuro Eval
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Puzzle Solving, Kakuro |
Overview
Implements the KakuroEvaluator for evaluating Kakuro (cross-sum) puzzle solutions in the MMHelix benchmark.
Description
The KakuroEvaluator extends BaseEvaluator to evaluate Kakuro puzzle solutions where empty cells are filled with digits 1-9 such that each horizontal/vertical group sums to its clue value with no repeated digits. The extract_answer method uses three strategies: answer block delimiters ([answer]/[/answer], <answer></answer>), common text delimiters (answer:/solution:/final answer:), and full text parsing. It returns a dictionary mapping (row, col) tuples to integer values. The evaluate method validates sum constraints and digit uniqueness within each group.
Usage
Called internally by the MMHelix dataset class during Kakuro puzzle evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/mmhelix/evaluators/kakuro_eval.py, Lines: L1-598 - Import:
from vlmeval.dataset.utils.mmhelix.evaluators.kakuro_eval import KakuroEvaluator
Key Functions:
class KakuroEvaluator(BaseEvaluator):
def extract_answer(self, model_output): ...
def evaluate(self, predicted_answer, ground_truth, initial_state): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Model output string containing cell-value mappings; ground-truth mappings; initial puzzle state with clue cells |
| Outputs | Boolean indicating whether the solution satisfies all Kakuro sum and uniqueness constraints |
Usage Examples
# Internal usage example
from vlmeval.dataset.utils.mmhelix.evaluators.kakuro_eval import KakuroEvaluator
evaluator = KakuroEvaluator()
answer = evaluator.extract_answer(model_output)
is_correct = evaluator.evaluate(answer, ground_truth, initial_state)