Implementation:EvolvingLMMs Lab Lmms eval SALBench Utils

Task utility functions for the SALBench (Spatial and Attribute Learning Benchmark), specifically the P3O3 tasks which evaluate object property understanding through set-based predictions.

Location

/tmp/kapso_repo_sslb_59s/lmms_eval/tasks/salbench/utils.py

Overview

Provides document processing, set-based result parsing, and hierarchical metric computation (exact match, precision, recall, F1) at sample, category, and per-category levels for P3 and O3 tasks.

Task Categories

P3 Categories

P3_CATEGORIES = ["orientation", "color", "size"]

O3 Categories

O3_CATEGORIES = ["orientation", "color", "focus", "shape", "size", "location", "pattern"]

Document Processing

p3o3_doc_to_visual(doc)

Extracts and converts image(s) to RGB

Parameters: doc - Document with image key

Process:

1. Returns None if no image
2. Handles both single image and list of images
3. Converts each to RGB
4. Returns None on conversion error (with warning)

Returns: List of RGB images, or None

p3o3_doc_to_text(doc, prompt_kwargs=None)

Constructs question prompt with optional pre/post prompts

Parameters:

• doc - Document with question key
• prompt_kwargs - Optional dict with pre_prompt and post_prompt

Process:

1. Raises ValueError if no question key
2. Strips question whitespace
3. If prompt_kwargs provided:
   • Validates presence of both pre_prompt and post_prompt
   • Formats as "{pre}\n{question}\n{post}"
4. Else: Appends default instruction

Returns: Formatted prompt string

Default Format: "{question}\nAnswer the question using a single word or phrase."

Result Processing

Set Parsing

Both P3 and O3 tasks expect comma-separated category predictions.

Parsing Logic:

• Strips leading/trailing [ and ]
• Splits on commas
• Lowercases and strips each element
• Creates set of predictions

Core Processing Function

process_results(doc, results, categories)

Computes sample-level and category-level metrics

Parameters:

• doc - Document with answer and image_id
• results - Model prediction list
• categories - List of valid category names

Process:

1. Parses prediction into set
2. Parses ground truth answer into set
3. Computes sample-level metrics:
   • Exact Match: int(pred == gt_ans)
   • Precision: len(matches) / (len(pred) + 1e-8)
   • Recall: len(matches) / len(gt_ans)
   • F1: 2 * precision * recall / (precision + recall + 1e-8)
4. For each category, computes confusion matrix:
   • True Positive: Category in both pred and ground truth
   • False Positive: Category in pred but not ground truth
   • True Negative: Category in neither
   • False Negative: Category in ground truth but not pred

Returns: Dictionary with metrics:

• exact_match: {"score": int}
• sample_precision: {"score": float}
• sample_recall: {"score": float}
• sample_f1: {"score": float}
• all_cat_precision, all_cat_recall, all_cat_f1: {"id": image_id, "pred": cat_preds}
• {cat}_precision, {cat}_recall, {cat}_f1: {"pred": cat_preds[cat]} (for each category)

Task-Specific Wrappers

p3_process_results(doc, results)

Processes P3 results

Returns: process_results(doc, results, P3_CATEGORIES)

o3_process_results(doc, results)

Processes O3 results

Returns: process_results(doc, results, O3_CATEGORIES)

Multiple Choice Variant

process_results_multiple_choices(doc, results)

Processes single-choice selection tasks

Parameters: doc, results

Assertion: Batch size must be 1

Process:

1. Extracts and lowercases prediction
2. Strips commas and parentheses
3. Takes first character (empty string if no response)
4. Compares with ground truth answer (lowercased)

Returns: Dictionary with acc entry: {"score": int}

Aggregation Functions

Sample-Level Metrics

aggregate_per_sample_score(results)

Computes mean of per-sample scores

Parameters: results - List of result dicts with score key

Returns: Average score (float)

Per-Category Metrics

_aggregate_per_category(results)

Aggregates single category's confusion matrix across all results

Parameters: results - List of result dicts with pred containing confusion matrix

Process:

1. Accumulates confusion matrix counts:
   • true_pos, true_neg, false_pos, false_neg
2. Computes metrics:
   • precision = true_pos / (true_pos + false_pos + 1e-8)
   • recall = true_pos / (true_pos + false_neg + 1e-8)
   • f1 = 2 * precision * recall / (precision + recall + 1e-8)

Returns: Tuple of (precision, recall, f1)

Wrapper Functions:

• aggregate_per_category_precision(results) - Returns precision
• aggregate_per_category_recall(results) - Returns recall
• aggregate_per_category_f1(results) - Returns F1

All-Category Metrics

_aggregate_all_category(results, categories)

Computes per-category metrics and averages across categories

Parameters:

• results - List of result dicts with pred containing category-level confusion matrices
• categories - List of category names

Process:

1. For each category:
   • Accumulates confusion matrix across all results
   • Computes precision, recall, F1
2. Computes aggregated metrics:
   • agg_precision = mean(precisions)
   • agg_recall = mean(recalls)
   • agg_f1 = mean(f1s)

Returns: Tuple of (agg_precision, agg_recall, agg_f1)

P3 Wrapper Functions:

• p3_aggregate_all_category_precision(results)
• p3_aggregate_all_category_recall(results)
• p3_aggregate_all_category_f1(results)

O3 Wrapper Functions:

• o3_aggregate_all_category_precision(results)
• o3_aggregate_all_category_recall(results)
• o3_aggregate_all_category_f1(results)

Metric Hierarchy

SALBench evaluates at three levels:

1. Sample Level:
   • Exact match of entire prediction set
   • Precision/recall/F1 of predicted elements
2. Per-Category Level:
   • Precision/recall/F1 for specific category (e.g., "color")
   • Based on binary classification (present vs. absent)
3. All-Category Level:
   • Average of per-category metrics
   • Macro-averaged across all categories

Epsilon Handling

All division operations add 1e-8 to denominators to avoid division by zero:

• precision = matches / (pred_count + 1e-8)
• recall = matches / (gt_count + 1e-8)
• f1 = 2 * p * r / (p + r + 1e-8)

Dependencies

None (pure Python implementation)

Design Notes

Set-Based Evaluation

• Predictions and ground truth are sets (order-independent)
• Exact match requires complete agreement
• Precision/recall/F1 measure partial credit

Category Binary Classification

• Each category treated as binary (present/not present)
• Enables fine-grained analysis of specific property understanding
• Supports both sample-level and category-level insights

Macro-Averaging

• All-category metrics use macro-averaging
• Treats each category equally regardless of frequency
• Provides balanced view of model capabilities

Related

• Task_Utility_Functions - General task utility pattern
• Set_Based_Evaluation - Set-based prediction evaluation
• Hierarchical_Metrics - Multi-level metric aggregation
• Confusion_Matrix_Metrics - Precision/recall/F1 from confusion matrices