Principle:Haotian liu LLaVA GPT Review Aggregation

Overview

Method for using GPT-4 as an automated judge to score and compare vision-language model outputs, with aggregation of per-question scores into category-level and overall performance metrics.

Description

GPT-4 review aggregation uses a two-stage process for qualitative evaluation of vision-language model outputs:

Stage 1: GPT-4 Scoring (eval_gpt_review_bench.py)

GPT-4 evaluates model outputs by comparing them against reference answers (typically GPT-4's own answers) using category-specific rubrics. For each question:

1. The evaluation prompt is constructed from the question context (image captions), the question text, two model answers (reference and candidate), and a category-specific evaluation rule
2. GPT-4 produces a review containing numerical scores for both the reference answer and the candidate answer
3. Scores are parsed as a tuple [reference_score, candidate_score] from the first line of GPT-4's response
4. Results are written to a review JSONL file with question_id, category, tuple, and the full review content

The evaluation categories for LLaVA-Bench include:

• llava_bench_conv - Conversational questions
• llava_bench_detail - Detailed description questions
• llava_bench_complex - Complex reasoning questions

Each category has its own evaluation prompt and role definition loaded from a rule JSON file (llava/eval/table/rule.json).

Stage 2: Score Aggregation (summarize_gpt_review.py)

The aggregation script reads GPT-4 review JSONL files and computes:

• Per-category mean scores - Average reference and candidate scores within each category
• Overall mean scores - Average across all questions regardless of category
• Relative performance - Computed as (candidate_score / reference_score) * 100

Usage

Use this for evaluating on LLaVA-Bench-in-the-Wild and other qualitative benchmarks where automated metrics like accuracy are insufficient. This approach captures nuanced quality differences in open-ended model responses.

Requirements:

• An OpenAI API key (set as environment variable)
• GPT-4 API access (uses gpt-4-0314 model)
• Reference answers (GPT-4 answers provided in answers_gpt4.jsonl)

End-to-End Pipeline (from llavabench.sh)

# Step 1: Generate model answers
python -m llava.eval.model_vqa \
    --model-path liuhaotian/llava-v1.5-13b \
    --question-file ./playground/data/eval/llava-bench-in-the-wild/questions.jsonl \
    --image-folder ./playground/data/eval/llava-bench-in-the-wild/images \
    --answers-file ./playground/data/eval/llava-bench-in-the-wild/answers/llava-v1.5-13b.jsonl \
    --temperature 0 \
    --conv-mode vicuna_v1

# Step 2: GPT-4 scoring
python llava/eval/eval_gpt_review_bench.py \
    --question playground/data/eval/llava-bench-in-the-wild/questions.jsonl \
    --context playground/data/eval/llava-bench-in-the-wild/context.jsonl \
    --rule llava/eval/table/rule.json \
    --answer-list \
        playground/data/eval/llava-bench-in-the-wild/answers_gpt4.jsonl \
        playground/data/eval/llava-bench-in-the-wild/answers/llava-v1.5-13b.jsonl \
    --output \
        playground/data/eval/llava-bench-in-the-wild/reviews/llava-v1.5-13b.jsonl

# Step 3: Score aggregation
python llava/eval/summarize_gpt_review.py \
    -f playground/data/eval/llava-bench-in-the-wild/reviews/llava-v1.5-13b.jsonl

Theoretical Basis

GPT-4-as-Judge Scoring

GPT-4 acts as an automated evaluator, assigning numerical ratings (typically 1-10) to model outputs. The system prompt instructs GPT-4 to be "a helpful and precise assistant for checking the quality of the answer." Category-specific evaluation rubrics are loaded from a rule file that defines:

• The evaluation prompt (scoring criteria and instructions)
• The role label for each answer (e.g., "Assistant" for conversational, "Description" for detail)

Relative Score Computation

Relative scores normalize the candidate model's performance against the reference:

relative_score = (candidate_score / reference_score) * 100

A score of 100 indicates parity with the reference (GPT-4); scores above 100 indicate the candidate outperforms the reference on average. The summary output reports:

category relative_score reference_score_x10 candidate_score_x10

Per-Category Aggregation

Scores within each category are aggregated using the arithmetic mean of score tuples. The numpy.asarray(v).mean(0) operation computes element-wise means across all score pairs in a category, yielding mean reference and mean candidate scores.

Resumability

The GPT-4 scoring script supports resumable evaluation: if the output file already exists, previously scored questions are skipped. This handles API rate limits and interruptions gracefully.

Knowledge Sources

• Paper - Visual Instruction Tuning - https://arxiv.org/abs/2304.08485
• Repo - LLaVA - https://github.com/haotian-liu/LLaVA

Domains

• Evaluation
• LLM_as_Judge

Related Pages

• Implementation:Haotian_liu_LLaVA_Summarize_GPT_Review

Metadata

Property	Value
last_updated	2026-02-13 14:00 GMT
page_type	Principle
workflow	Benchmark_Evaluation