Implementation:Open compass VLMEvalKit MMHelix Campsite Eval

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Puzzle Solving, Campsite

Overview

Implements the CampsiteEvaluator for evaluating tent placement puzzle solutions in the MMHelix benchmark.

Description

The CampsiteEvaluator validates tent placement solutions where tents must be orthogonally adjacent to trees, not adjacent to each other (even diagonally), and satisfy row/column count constraints. The extract_answer method uses five extraction strategies with decreasing priority: direct parsing, regex patterns, number pairing, special formats, and aggressive extraction. Coordinates use 1-based indexing with the top-left corner at [1,1]. The evaluate method verifies all three campsite puzzle rules against the initial grid state.

Usage

Called internally by the MMHelix dataset class during Campsite puzzle evaluation.

Code Reference

• Source: vlmeval/dataset/utils/mmhelix/evaluators/campsite_eval.py, Lines: L1-595
• Import: from vlmeval.dataset.utils.mmhelix.evaluators.campsite_eval import CampsiteEvaluator

Key Functions:

class CampsiteEvaluator:
    def __init__(self): ...
    def extract_answer(self, model_output): ...
    def evaluate(self, predicted_answer, ground_truth, initial_state): ...

I/O Contract

Direction	Description
Inputs	Model output string containing tent coordinates; ground-truth tent positions; initial grid with trees and row/column constraints
Outputs	Boolean indicating whether the tent placement satisfies all campsite puzzle rules

Usage Examples

# Internal usage example
from vlmeval.dataset.utils.mmhelix.evaluators.campsite_eval import CampsiteEvaluator
evaluator = CampsiteEvaluator()
tents = evaluator.extract_answer(model_output)
is_correct = evaluator.evaluate(tents, ground_truth, initial_state)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction