Implementation:Open compass VLMEvalKit MMHelix Base Evaluator

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Puzzle Solving, Base Classes

Overview

Defines base evaluator classes for the MMHelix puzzle-solving benchmark, providing common interfaces for answer extraction, evaluation, and string matching.

Description

This module provides three base classes: `BaseEvaluator` with abstract methods `prepare_prompt`, `extract_answer`, and `evaluate`; `SimpleStrMatch` for whitespace-normalized case-insensitive string comparison; and `MatchFromList` for matching against multiple acceptable answers. The `BaseEvaluator.extract_answer` method includes a default implementation that extracts content from `<answer></answer>` tags in model output, with fallback to the full output text.

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/mmhelix/evaluator.py, Lines: L1-53
• Import: from vlmeval.dataset.utils.mmhelix.evaluator import BaseEvaluator, SimpleStrMatch, MatchFromList

Key Functions:

class BaseEvaluator:
    def extract_answer(self, model_output: str) -> Any: ...
    def evaluate(self, predicted_answer, ground_truth, initial_state) -> bool: ...

class SimpleStrMatch(BaseEvaluator): ...
class MatchFromList(BaseEvaluator): ...

I/O Contract

Direction	Description
Inputs	Model output strings; ground truth answers (string or list)
Outputs	Extracted answer strings; boolean evaluation results

Usage Examples

from vlmeval.dataset.utils.mmhelix.evaluator import SimpleStrMatch

evaluator = SimpleStrMatch()
result = evaluator.evaluate("42", "42", None)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction