Implementation:Open compass VLMEvalKit MMHelix Skyscrapers Eval

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Puzzle Solving, Skyscrapers

Overview

Evaluates Skyscrapers puzzle solutions in the MMHelix benchmark by verifying row/column uniqueness and visibility constraints from four directions.

Description

The `SkyscrapersEvaluator` class extends `BaseEvaluator` to validate Skyscrapers puzzle solutions. It verifies two constraint types: each row and column must contain numbers 1 to n exactly once (Latin square), and the visibility clues from each edge must match (taller buildings block the view of shorter ones behind them). The `extract_answer` method parses 2D integer arrays from model output using regex and JSON parsing, with robust validation of array structure.

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/mmhelix/evaluators/skyscrapers_evaluator.py, Lines: L1-230
• Import: from vlmeval.dataset.utils.mmhelix.evaluators.skyscrapers_evaluator import SkyscrapersEvaluator

Key Functions:

class SkyscrapersEvaluator(BaseEvaluator):
    def extract_answer(self, model_output) -> List[List[int]]: ...
    def evaluate(self, predicted_answer, ground_truth, params) -> bool: ...

I/O Contract

Direction	Description
Inputs	Model output string with a 2D grid; puzzle params with edge visibility clues
Outputs	Boolean indicating whether the solution satisfies Latin square and visibility constraints

Usage Examples

from vlmeval.dataset.utils.mmhelix.evaluators.skyscrapers_evaluator import SkyscrapersEvaluator

evaluator = SkyscrapersEvaluator()
grid = evaluator.extract_answer(model_output)
is_correct = evaluator.evaluate(grid, ground_truth, params)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction