Implementation:Open compass VLMEvalKit MMHelix Cryptomath Eval

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Puzzle Solving, Cryptarithmetic

Overview

Evaluates cryptarithmetic (CryptoMath) puzzle solutions in the MMHelix benchmark by verifying letter-to-digit mappings satisfy the given equation.

Description

The `CryptoMathEvaluator` class extends `BaseEvaluator` to validate solutions to letter arithmetic puzzles. It verifies that each letter maps to a unique digit (0-9), that the equation is satisfied when letters are replaced with digits, and that leading letters are not mapped to zero. The `extract_answer` method uses multiple parsing strategies (ast.literal_eval, JSON, regex) to extract letter-to-digit mapping dictionaries from model output.

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/mmhelix/evaluators/cryptomath_eval.py, Lines: L1-196
• Import: from vlmeval.dataset.utils.mmhelix.evaluators.cryptomath_eval import CryptoMathEvaluator

Key Functions:

class CryptoMathEvaluator(BaseEvaluator):
    def extract_answer(self, model_output) -> Dict[str, int]: ...
    def evaluate(self, predicted_answer, ground_truth, params) -> bool: ...

I/O Contract

Direction	Description
Inputs	Model output string containing a letter-to-digit mapping; puzzle equation parameters
Outputs	Boolean indicating whether the mapping satisfies the cryptarithmetic equation

Usage Examples

from vlmeval.dataset.utils.mmhelix.evaluators.cryptomath_eval import CryptoMathEvaluator

evaluator = CryptoMathEvaluator()
mapping = evaluator.extract_answer(model_output)
is_correct = evaluator.evaluate(mapping, ground_truth, params)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction