Implementation:Open compass VLMEvalKit MMHelix Eulero Eval

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Puzzle Solving, Eulero

Overview

Implements the EuleroEvaluator for evaluating Eulero (Graeco-Latin square) puzzle solutions in the MMHelix benchmark.

Description

The EuleroEvaluator extends BaseEvaluator to evaluate Eulero puzzle solutions where each cell contains a letter-number pair (e.g., A1, B2) such that every row and column has unique letters and unique numbers. The extract_answer method prioritizes extraction from <answer> tags, then normalizes pipe-delimited grid text, and falls back to extracting A1/B2-style pairs from free text. The evaluate method validates uniqueness constraints for both letter and number components across rows and columns.

Usage

Called internally by the MMHelix dataset class during Eulero puzzle evaluation.

Code Reference

• Source: vlmeval/dataset/utils/mmhelix/evaluators/eulero_eval.py, Lines: L1-348
• Import: from vlmeval.dataset.utils.mmhelix.evaluators.eulero_eval import EuleroEvaluator

Key Functions:

class EuleroEvaluator(BaseEvaluator):
    def prepare_prompt(self, question, params=None): ...
    def extract_answer(self, model_output): ...
    def evaluate(self, predicted_answer, ground_truth, initial_state, params=None): ...

I/O Contract

Direction	Description
Inputs	Model output string containing a grid of letter-number pairs; ground-truth grid; initial puzzle state
Outputs	Boolean indicating whether the Eulero solution satisfies all uniqueness constraints

Usage Examples

# Internal usage example
from vlmeval.dataset.utils.mmhelix.evaluators.eulero_eval import EuleroEvaluator
evaluator = EuleroEvaluator()
grid = evaluator.extract_answer(model_output)
is_correct = evaluator.evaluate(grid, ground_truth, initial_state)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction