Implementation:Open compass VLMEvalKit MMHelix TwentyFourPoints Eval

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Puzzle Solving, 24 Game

Overview

Evaluates 24-point game solutions in the MMHelix benchmark by verifying that each given number is used exactly once and the expression evaluates to 24.

Description

The `TwentyFourPointsEvaluator` class extends `BaseEvaluator` to validate 24-point game solutions. The `extract_answer` method handles complex expression parsing including LaTeX symbols (\\times, \\div, \\cdot), Unicode operators, and various notation formats. It normalizes mathematical operators and extracts the longest valid expression from model output. Evaluation verifies that all given numbers appear exactly once and the final computation equals 24.

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/mmhelix/evaluators/twentyfourpoints_evaluator.py, Lines: L1-243
• Import: from vlmeval.dataset.utils.mmhelix.evaluators.twentyfourpoints_evaluator import TwentyFourPointsEvaluator

Key Functions:

class TwentyFourPointsEvaluator(BaseEvaluator):
    def prepare_prompt(self, question, params): ...
    def extract_answer(self, model_output) -> str: ...
    def evaluate(self, predicted_answer, ground_truth, params) -> bool: ...

I/O Contract

Direction	Description
Inputs	Model output string containing a mathematical expression; given numbers for the 24 game
Outputs	Boolean indicating whether the expression uses all numbers once and equals 24

Usage Examples

from vlmeval.dataset.utils.mmhelix.evaluators.twentyfourpoints_evaluator import TwentyFourPointsEvaluator

evaluator = TwentyFourPointsEvaluator()
expr = evaluator.extract_answer(model_output)
is_correct = evaluator.evaluate(expr, ground_truth, params)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction