Implementation:Open compass VLMEvalKit ChartMimic Color Evaluator
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Chart Understanding, Color Accuracy |
Overview
Implements the ColorEvaluator class for evaluating color accuracy in chart reproduction for the ChartMimic benchmark.
Description
This module provides the ColorEvaluator class that measures how accurately model-generated charts reproduce the colors of reference charts. It computes color similarity using permutation-based matching through calculate_similarity_for_permutation, with multiprocessing support for efficient permutation evaluation. Colors are grouped using the group_color utility, and similarity is computed per-color using calculate_similarity_single from the CIE76 color difference metric in CIELAB space. The evaluator handles different color formats and supports parallel computation via Python multiprocessing.
Usage
Called internally by the ChartMimic evaluator during color accuracy assessment.
Code Reference
- Source:
vlmeval/dataset/utils/chartmimic/evaluator/color_evaluator.py, Lines: L1-326 - Import:
from vlmeval.dataset.utils.chartmimic.evaluator.color_evaluator import ColorEvaluator
Key Functions:
def calculate_similarity_for_permutation(args): ...
class ColorEvaluator:
def __init__(self) -> None: ...
def evaluate(self, pred_colors, gt_colors): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Lists of predicted and ground-truth color values (hex strings or color names) extracted from chart rendering |
| Outputs | Color similarity score between 0.0 and 1.0 based on optimal permutation matching in CIELAB space |
Usage Examples
# Internal usage example
from vlmeval.dataset.utils.chartmimic.evaluator.color_evaluator import ColorEvaluator
evaluator = ColorEvaluator()
score = evaluator.evaluate(pred_colors, gt_colors)