Implementation:EvolvingLMMs Lab Lmms eval DetailCaps Utils

File: lmms_eval/tasks/detailcaps/utils.py (198 lines)

Principle: Task Utility Functions

Overview

Utility functions for the DetailCaps image captioning evaluation task. Evaluates generated captions against multiple reference captions using standard captioning metrics: BLEU (1-4), METEOR, ROUGE_L, CIDEr, and CAPTURE. Uses COCO evaluation tools for consistent metric calculation.

Constants

• detailcaps_METRICS - List of evaluated metrics:
  • CAPTURE, Bleu_4, Bleu_3, Bleu_2, Bleu_1, METEOR, ROUGE_L, CIDEr

Key Functions

detailcaps_doc_to_visual

def detailcaps_doc_to_visual(doc)

Converts binary image data to PIL Image.

Parameters:

• doc - Document with "binary" field containing image bytes

Returns: List with single RGB PIL Image

detailcaps_doc_to_text

def detailcaps_doc_to_text(doc, lmms_eval_specific_kwargs=None)

Returns captioning prompt from kwargs.

Parameters:

• lmms_eval_specific_kwargs - Dictionary with "prompt" field

Returns: Prompt string (e.g., "Please carefully observe the image and come up with a caption for the image")

detailcaps_doc_to_target

def detailcaps_doc_to_target(doc)

Extracts reference captions from three sources.

Returns: List of three reference captions:

• GT_Caption_GPT4O
• GT_Caption_GPT4V
• GT_Caption_Gemini15Pro

detailcaps_process_result

def detailcaps_process_result(doc, result)

Processes single result for metric evaluation.

Parameters:

• doc - Document with "image" field (used as image_id)
• result - Model's generated caption

Returns: Dictionary mapping each metric to data_dict:

{
  "detailcaps_CAPTURE": {answer, pred, image_id},
  "detailcaps_Bleu_4": {answer, pred, image_id},
  ...
}

detailcaps_aggregation_result

def detailcaps_aggregation_result(results, metric, args=None)

Core aggregation function for computing metric scores.

Parameters:

• results - List of processed results
• metric - Metric name to compute
• args - Optional arguments for file generation

Returns: Metric score (float)

Processing Steps:

1. Build COCO Dataset:
   • Creates "annotations" list with all reference captions
   • Creates "images" list with image IDs
   • Each annotation gets unique ID

1. Create COCO Objects:
   • Initializes COCO object with dataset
   • Loads results using coco.loadRes()
   • Creates COCOEvalCap evaluator

1. Prepare Predictions and References:
   • Organizes by image ID
   • gts: Ground truth captions
   • res: Model predictions

1. Tokenization:
   • Uses PTBTokenizer for BLEU, METEOR, ROUGE_L, CIDEr
   • CAPTURE uses raw text (reorganizes into lists)

1. Compute Score:
   • Calls appropriate scorer's compute_score method
   • For BLEU metrics, extracts specific n-gram score

1. Save Results:
   • Generates submission file with predictions
   • Format: [{"image_id": ..., "caption": ...}, ...]

Scorers Dictionary:

scorers = [
    (Bleu(4), "Bleu_1"),
    (Bleu(4), "Bleu_2"),
    (Bleu(4), "Bleu_3"),
    (Bleu(4), "Bleu_4"),
    (Meteor(), "METEOR"),
    (Rouge(), "ROUGE_L"),
    (Cider(), "CIDEr"),
    (CAPTURE(), "CAPTURE")
]

Metric-Specific Aggregation Functions

All follow the same pattern, calling detailcaps_aggregation_result with specific metric:

detailcaps_bleu4

def detailcaps_bleu4(results, args=None)

Computes BLEU-4 score.

detailcaps_bleu3

Computes BLEU-3 score.

detailcaps_bleu2

Computes BLEU-2 score.

detailcaps_bleu1

Computes BLEU-1 score.

detailcaps_meteor

Computes METEOR score.

detailcaps_rougel

Computes ROUGE_L score.

detailcaps_cider

Computes CIDEr score.

detailcaps_spice

Computes SPICE score (commented in metrics list).

detailcaps_capture

Computes CAPTURE score.

Test Set Functions

detailcaps_test_process_result

def detailcaps_test_process_result(doc, result)

Processes test results without evaluation.

Returns: Dictionary with "detailcaps_passthrough" containing pred and image_id

detailcaps_test_aggregation_result

def detailcaps_test_aggregation_result(results, args=None)

Generates test submission file.

Output Format:

[
  {"image_id": <int>, "caption": "<string>"},
  ...
]

File Name: detailcaps_captions_detailcaps_test_alg_results.json

Note: Logs reminder to also submit validation results to CodaLab

Usage Pattern

Validation Set:

1. Convert binary image to RGB PIL Image
2. Generate caption using prompt
3. Compare against 3 reference captions (GPT-4O, GPT-4V, Gemini-1.5-Pro)
4. Tokenize using PTBTokenizer (except CAPTURE)
5. Compute 8 metrics using COCO evaluation tools
6. Save results to submission file

Test Set:

1. Generate captions
2. Save to submission file for server evaluation
3. No local metrics computed

Dependencies

• collections, io, json, logging, os - Standard library
• PIL.Image - Image loading
• pycocotools.coco.COCO - COCO dataset handling
• pycocoevalcap.eval.COCOEvalCap - Evaluation framework
• pycocoevalcap.eval - Bleu, Cider, Meteor, Rouge metrics
• pycocoevalcap.tokenizer.ptbtokenizer.PTBTokenizer - Tokenization
• capture_metric.capture.CAPTURE - CAPTURE metric
• lmms_eval.tasks._task_utils.file_utils - Submission file generation

COCO Dataset Format

Annotations:

{
  "image_id": <string>,
  "caption": <string>,
  "id": <int>
}

Images:

{
  "id": <string>
}

Results (Predictions):

{
  "image_id": <string>,
  "caption": <string>
}