Implementation:Haotian liu LLaVA Summarize GPT Review

Overview

CLI script for aggregating GPT-4 review scores across evaluation categories, computing per-category and overall relative performance metrics for LLaVA-Bench and similar qualitative benchmarks.

Description

summarize_gpt_review.py reads GPT-4 review JSONL files (produced by eval_gpt_review_bench.py), parses score tuples, and computes per-category and overall relative scores. The script supports filtering by GPT version, experiment name, and question IDs to ignore.

The aggregation process:

1. Reads each review line from the JSONL file
2. Filters out ignored question IDs (specified via -i)
3. Groups score tuples by category field (e.g., llava_bench_conv, llava_bench_detail, llava_bench_complex)
4. Adds each score to both its category bucket and the all bucket
5. Computes element-wise mean of score tuples per category via numpy.asarray(v).mean(0)
6. Prints relative score as (candidate_mean / reference_mean) * 100

The script also handles review files without category fields (used in older evaluation formats) and can process either individual files (via -f) or entire directories of review files (via -d).

Source

llava/eval/summarize_gpt_review.py:L9-60

CLI Signature

python llava/eval/summarize_gpt_review.py \
    -d /path/to/review/dir \
    -v gpt4 \
    -s experiment_name \
    -f specific_review_file.jsonl \
    -i 5 12 27

Argument	Long Form	Type	Default	Description
-d	--dir	str	None	Directory containing review JSONL files
-v	--version	str	None	Filter by GPT version (e.g., 0613 or 0314)
-s	--select	list	None	Filter: only process configs containing all specified substrings
-f	--files	list	[]	Explicit list of review file paths to process
-i	--ignore	list	[]	Question IDs to exclude from aggregation

File Discovery Logic

When using -d (directory mode), the script auto-discovers review files matching these filename patterns:

• gpt4_text_*.jsonl
• reviews_*.jsonl
• review_*.jsonl
• Any .jsonl file if the directory path contains "review"

When using -f (file mode), the specified files are used directly.

Version Detection

The GPT version is extracted from the config filename: files containing 0613 are tagged as version 0613, others default to 0314. The -v flag filters to only process matching versions.

Inputs

GPT-4 review JSONL files with one JSON object per line:

{
    "id": 1,
    "question_id": 42,
    "answer1_id": "gpt4_answer_id",
    "answer2_id": "model_answer_uuid",
    "category": "llava_bench_conv",
    "content": "8 7\nThe first assistant provides a more detailed...",
    "tuple": [8.0, 7.0]
}

The tuple field contains [reference_score, candidate_score] where scores are typically on a 1-10 scale. If no tuple field is present, the script falls back to a score field.

Outputs

Per-category and overall relative scores printed to stdout:

llava-v1.5-13b
llava_bench_complex 85.2 7.8 6.6
llava_bench_conv 92.1 8.1 7.5
llava_bench_detail 78.3 7.5 5.9
all 85.2 7.8 6.6
=================================

Each line format: category relative_score reference_mean_x10 candidate_mean_x10

Where:

• relative_score = round(candidate_mean / reference_mean * 100, 1)
• reference_mean_x10 = round(reference_mean * 10, 1)
• candidate_mean_x10 = round(candidate_mean * 10, 1)

The multiplication by 10 scales scores from the 0-10 range to 0-100 for readability.

Usage Example

From llavabench.sh (End-to-End)

# After generating model answers and running GPT-4 review:
python llava/eval/summarize_gpt_review.py \
    -f playground/data/eval/llava-bench-in-the-wild/reviews/llava-v1.5-13b.jsonl

Batch Processing a Directory of Reviews

python llava/eval/summarize_gpt_review.py \
    -d playground/data/eval/llava-bench-in-the-wild/reviews/ \
    -v 0314

Ignoring Specific Questions

python llava/eval/summarize_gpt_review.py \
    -f reviews/llava-v1.5-13b.jsonl \
    -i 5 12 27

Key Implementation Details

• Score parsing is performed by eval_gpt_review_bench.py's parse_score() function, which extracts two floats from the first line of GPT-4's review (separated by space or comma). Invalid parses return [-1, -1].
• Rate limit handling in the GPT-4 scoring script uses a retry loop with 0.5 second sleep between retries on RateLimitError.
• Resumable evaluation is supported: if the output review file already exists, previously scored questions are loaded and skipped during re-runs.
• The context file provides image captions and descriptions used to ground GPT-4's evaluation, since GPT-4 cannot directly view images.

Related Pages

• implements Principle:Haotian_liu_LLaVA_GPT_Review_Aggregation

Metadata

Property	Value
last_updated	2026-02-13 14:00 GMT
page_type	Implementation (API Doc)
workflow	Benchmark_Evaluation
source_file	llava/eval/summarize_gpt_review.py

• Environment:Haotian_liu_LLaVA_OpenAI_API_Evaluation_Environment