Implementation:Open compass VLMEvalKit MMHelix WordLadder Eval

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Puzzle Solving, Word Ladder

Overview

Evaluates Word Ladder puzzle solutions in the MMHelix benchmark by verifying single-letter changes, word validity, and path completeness.

Description

The `WordLadderEvaluator` class extends `BaseEvaluator` to validate Word Ladder solutions where each step must change exactly one letter and produce a valid English word. It uses NLTK's `words` corpus for dictionary validation. The evaluator checks that the path starts with the source word, ends with the target word, each consecutive pair differs by exactly one letter, all intermediate words are valid English words, and the path length is within the specified range.

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/mmhelix/evaluators/wordladder_eval.py, Lines: L1-272
• Import: from vlmeval.dataset.utils.mmhelix.evaluators.wordladder_eval import WordLadderEvaluator

Key Functions:

class WordLadderEvaluator(BaseEvaluator):
    def prepare_prompt(self, question, params): ...
    def extract_answer(self, model_output) -> List[str]: ...
    def evaluate(self, predicted_answer, ground_truth, params) -> bool: ...

I/O Contract

Direction	Description
Inputs	Model output string with a word sequence; puzzle params with start_word, target_word, and solution length
Outputs	Boolean indicating whether the word ladder is valid (single-letter changes, valid words, correct endpoints)

Usage Examples

from vlmeval.dataset.utils.mmhelix.evaluators.wordladder_eval import WordLadderEvaluator

evaluator = WordLadderEvaluator()
words = evaluator.extract_answer(model_output)
is_correct = evaluator.evaluate(words, ground_truth, params)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction