Implementation:Open compass VLMEvalKit LLaVABench Utils
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, LLM Judge, Conversational |
Overview
Provides GPT-based pairwise evaluation utilities for the LLaVA-Bench benchmark, comparing model responses against GPT-4 reference answers.
Description
This module defines evaluation rules (`rule_dict`) for three LLaVA-Bench categories (conv, detail, complex) with scoring prompts for helpfulness, relevance, accuracy, and detail level on a 1-10 scale. The `build_prompt` function constructs pairwise comparison prompts, and `parse_score` extracts score pairs from the judge model's output. It includes Korean language support via `rule_dict_ko` and `build_prompt_ko` for multilingual evaluation.
Usage
Called internally by the corresponding dataset class during evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/llavabench.py, Lines: L1-88 - Import:
from vlmeval.dataset.utils.llavabench import build_prompt, parse_score
Key Functions:
def build_prompt(line): ...
def parse_score(review): ...
def get_eval(judge, content): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | A data line dict with 'caption', 'question', 'gpt4_ans', 'prediction', and 'category' fields |
| Outputs | Formatted evaluation prompt string; parsed score list [score1, score2] from judge response |
Usage Examples
from vlmeval.dataset.utils.llavabench import build_prompt, parse_score
prompt = build_prompt(line)
scores = parse_score("8 7\nExplanation...")
# scores = [8.0, 7.0]
Related Pages
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment