Implementation:Open compass VLMEvalKit MMHelix Nibbles Eval

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Puzzle Solving, Snake Game

Overview

Evaluates Snake (Nibbles) game solutions in the MMHelix benchmark by simulating game movement and verifying apple collection.

Description

The `NibblesEvaluator` class extends `BaseEvaluator` to validate Snake game solutions by simulating the game from an initial state. The `extract_answer` method parses movement directions (up, down, left, right) from model output, including extraction from `<answer>` tags. The `evaluate` method simulates the game step by step, tracking the snake's position, checking for collisions with walls/body, and verifying that all apples are collected via the specified movement sequence.

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/mmhelix/evaluators/nibbles_eval.py, Lines: L1-264
• Import: from vlmeval.dataset.utils.mmhelix.evaluators.nibbles_eval import NibblesEvaluator

Key Functions:

class NibblesEvaluator(BaseEvaluator):
    def extract_answer(self, model_output) -> str: ...
    def evaluate(self, predicted_answer, ground_truth, initial_state) -> bool: ...

I/O Contract

Direction	Description
Inputs	Model output with movement directions; initial game state with grid size, snake position, and apple locations
Outputs	Boolean indicating whether the movement sequence successfully collects all apples without collisions

Usage Examples

from vlmeval.dataset.utils.mmhelix.evaluators.nibbles_eval import NibblesEvaluator

evaluator = NibblesEvaluator()
moves = evaluator.extract_answer(model_output)
is_correct = evaluator.evaluate(moves, ground_truth, initial_state)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction