Implementation:Open compass VLMEvalKit MMHELIX

Field	Value
source	VLMEvalKit
domain	Vision, Benchmarking, Puzzles, Graph Problems, Algorithms

Overview

Benchmark dataset implementation for MM-HELIX multi-modal puzzle and reasoning evaluation in VLMEvalKit.

Description

MMHELIX inherits from ImageBaseDataset and implements the MM-HELIX benchmark for evaluating multimodal reasoning across diverse task categories. The TYPE field is set to 'VQA'. It covers four major groups: graph_problems (connectivity, Eulerian/Hamiltonian paths, topological sort), puzzles (Sudoku, Nonogram, Kakuro, etc.), algorithm_problems (24Points, CryptoMath, etc.), and games (Sokoban, Minesweeper, Maze, etc.), with both standard and language variants.

Usage

Registered in vlmeval/dataset/__init__.py and invoked through build_dataset() by benchmark name.

Code Reference

Source: vlmeval/dataset/mmhelix.py, Lines: L1-579
Import: from vlmeval.dataset.mmhelix import MMHELIX

Signature:

class MMHELIX(ImageBaseDataset):
    TYPE = 'VQA'
    DATASET_URL = {...}
    GROUP_LIST = {...}
    ...

I/O Contract

Direction	Description
Inputs	TSV dataset file with puzzle/reasoning images and questions
Outputs	Evaluation results DataFrame with scores per task group

Usage Examples

from vlmeval.dataset import build_dataset
dataset = build_dataset('MM-HELIX')

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment