Implementation:Open compass VLMEvalKit ChartMimic Grid Evaluator

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Chart Generation, Grid Layout

Overview

Evaluates grid/legend layout accuracy by comparing grid configurations extracted from generated and golden matplotlib code in the ChartMimic benchmark.

Description

The `GridEvaluator` class instruments matplotlib code to log grid and legend objects. It injects code to capture grid properties, executes the modified scripts safely, and compares grid configurations between generated and reference charts. Evaluation uses precision, recall, and F1 metrics over detected grid elements to measure how accurately the generated chart reproduces the reference grid layout.

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/chartmimic/evaluator/grid_evaluator.py, Lines: L1-181
• Import: from vlmeval.dataset.utils.chartmimic.evaluator.grid_evaluator import GridEvaluator

Key Functions:

class GridEvaluator:
    def __call__(self, generation_code_file, golden_code_file): ...
    def _log_legends(self, code_file): ...
    def _calculate_metrics(self, generation_grids, golden_grids): ...

I/O Contract

Direction	Description
Inputs	Paths to generated and golden Python code files producing matplotlib charts
Outputs	Metrics dict with precision, recall, and F1 scores for grid element matching

Usage Examples

from vlmeval.dataset.utils.chartmimic.evaluator.grid_evaluator import GridEvaluator

evaluator = GridEvaluator()
evaluator("generated_chart.py", "golden_chart.py")
print(evaluator.metrics)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction