Implementation:Open compass VLMEvalKit MEGABench VLM As Judge

Field	Value
source	VLMEvalKit
domain	Vision, Evaluation, LLM Judge, Open-ended Generation

Overview

Implements a VLM (Vision-Language Model) as judge for evaluating open-ended generation tasks in the MEGA-Bench framework using GPT-4o.

Description

The `OpenAIVLMJudger` abstract class provides infrastructure for using GPT-4o (or configurable models) as an evaluation judge for open-ended generation tasks. It supports both text-only and multimodal evaluation with image encoding (base64), automatic image resizing, and configurable evaluation prompts from metric configurations. The class manages OpenAI API interactions, template mapping for prompt construction, and reference type handling (with/without reference answers).

Usage

Called internally by the corresponding dataset class during evaluation.

Code Reference

• Source: vlmeval/dataset/utils/megabench/scoring/vlm_as_judge.py, Lines: L1-236
• Import: from vlmeval.dataset.utils.megabench.scoring.vlm_as_judge import OpenAIVLMJudger

Key Functions:

class OpenAIVLMJudger(abc.ABC):
    def __init__(self, metric_config, model="gpt-4o-2024-08-06", ...): ...
    def create_image_content(self, image_path): ...
    def encode_image(self, image_path): ...

I/O Contract

Direction	Description
Inputs	Metric configuration dict with eval_criteria_prompt, reference_type, and template_mapping; image paths and response text
Outputs	Judge evaluation scores from GPT-4o API calls

Usage Examples

from vlmeval.dataset.utils.megabench.scoring.vlm_as_judge import OpenAIVLMJudger

# OpenAIVLMJudger is abstract; use a concrete subclass

Related Pages

• Principle:Open_compass_VLMEvalKit_Benchmark_Dataset_Construction