Implementation:Open compass VLMEvalKit MMSci Utils
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Scientific Figure Captioning |
Overview
Provides evaluation utilities for the MMSci benchmark, including CIDEr scoring, G-Eval relevance scoring, and atomic fact generation for scientific figure caption evaluation.
Description
This module implements evaluation metrics for scientific figure captioning quality. It includes compute_cider using the pycocoevalcap CIDEr scorer for caption similarity, g_eval_prompt for LLM-based relevance scoring on a 1-5 scale, and generate_atomic_facts_sys_prompt for breaking down multi-panel figure captions into independent atomic facts. The evaluation pipeline combines automated metrics (CIDEr, BLEU, BERTScore) with LLM-as-judge approaches using the HuggingFace evaluate library for comprehensive caption quality assessment.
Usage
Called internally by the MMSci dataset class during scientific figure caption evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/mmsci.py, Lines: L1-434 - Import:
from vlmeval.dataset.utils.mmsci import compute_cider
Key Functions:
def compute_cider(predictions, references): ...
# Prompt templates
g_eval_prompt = """..."""
generate_atomic_facts_sys_prompt = """..."""
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Lists of predicted caption strings and reference caption strings; multi-panel figure descriptions for atomic fact extraction |
| Outputs | CIDEr scores as floats; relevance scores (1-5); lists of atomic fact strings |
Usage Examples
# Internal usage example
from vlmeval.dataset.utils.mmsci import compute_cider
score = compute_cider(["predicted caption"], [["reference caption 1", "reference caption 2"]])