Implementation:Open compass VLMEvalKit ChartMimic Layout Evaluator

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, Chart Generation, Layout

Overview

Evaluates subplot and axes layout accuracy by comparing layout configurations extracted from generated and golden matplotlib code in the ChartMimic benchmark.

Description

The `LayoutEvaluator` class instruments matplotlib code to log layout and axes objects including subplot arrangements. It handles special cases for graph-type charts with a dedicated suffix method. The evaluator executes modified scripts via `run_script_safe`, reads layout output files, and computes precision, recall, and F1 metrics comparing generated versus reference chart layouts.

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/chartmimic/evaluator/layout_evaluator.py, Lines: L1-166
• Import: from vlmeval.dataset.utils.chartmimic.evaluator.layout_evaluator import LayoutEvaluator

Key Functions:

class LayoutEvaluator:
    def __call__(self, generation_code_file, golden_code_file): ...
    def _log_layouts(self, code_file): ...
    def _calculate_metrics(self, generation_layouts, golden_layouts): ...

I/O Contract

Direction	Description
Inputs	Paths to generated and golden Python code files producing matplotlib charts
Outputs	Metrics dict with precision, recall, and F1 scores for layout matching

Usage Examples

from vlmeval.dataset.utils.chartmimic.evaluator.layout_evaluator import LayoutEvaluator

evaluator = LayoutEvaluator()
evaluator("generated_chart.py", "golden_chart.py")
print(evaluator.metrics)

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction