Implementation:Open compass VLMEvalKit MMHelix Binario Eval

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Puzzle Solving, Binario

Overview

Implements the BinarioEvaluator for evaluating Binairo (binary puzzle) solutions in the MMHelix benchmark.

Description

The BinarioEvaluator extends BaseEvaluator to evaluate matrix-based binary puzzle solutions where grids must be filled with 0s and 1s following specific constraints. The extract_answer method uses multiple extraction strategies ordered by preference: literal newlines, code blocks, answer sections, solution sections, result sections, and matrix keyword detection. The evaluator validates that each row and column contains equal numbers of 0s and 1s, no three consecutive identical values exist, and all rows/columns are unique.

Usage

Called internally by the MMHelix dataset class during Binario puzzle evaluation.

Code Reference

• Source: vlmeval/dataset/utils/mmhelix/evaluators/binario_eval.py, Lines: L1-1012
• Import: from vlmeval.dataset.utils.mmhelix.evaluators.binario_eval import BinarioEvaluator

Key Functions:

class BinarioEvaluator(BaseEvaluator):
    def prepare_prompt(self, question, params): ...
    def extract_answer(self, model_output): ...
    def evaluate(self, predicted_answer, ground_truth=None, initial_state=None): ...

I/O Contract

Direction	Description
Inputs	Model output string containing a matrix of 0s and 1s; optional ground-truth matrix and initial state
Outputs	Boolean indicating whether the predicted Binario solution satisfies all puzzle constraints

Usage Examples

# Internal usage example
from vlmeval.dataset.utils.mmhelix.evaluators.binario_eval import BinarioEvaluator
evaluator = BinarioEvaluator()
matrix = evaluator.extract_answer(model_output)
is_correct = evaluator.evaluate(matrix)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction