Implementation:EvolvingLMMs Lab Lmms eval MMSearch Plus Utils

Location: /tmp/kapso_repo_sslb_59s/lmms_eval/tasks/mmsearch_plus/utils.py

Principle: Task_Utility_Functions

Purpose

Task-specific utilities for MMSearch-Plus VQA (Visual Question Answering) benchmark with decryption support and F1/exact match metrics.

Constants

• CANARY = "MMSearch-Plus" - decryption password

Key Functions

mmsearch_plus_doc_to_visual

def mmsearch_plus_doc_to_visual(doc: Dict[str, Any]) -> List[Any]

Extracts images from document:

• Decrypts document using decrypt_sample()
• Reads num_images field
• Extracts img_1, img_2, ... img_N fields
• Converts each to RGB format
• Logs warnings for missing/failed images
• Returns list of PIL Images

mmsearch_plus_doc_to_text

def mmsearch_plus_doc_to_text(doc: Dict[str, Any], lmms_eval_specific_kwargs: Dict[str, Any] = None) -> str

Formats question with prompts:

• Decrypts document
• Applies pre_prompt from kwargs (default: "")
• Applies post_prompt from kwargs (default: "\\nAnswer the question using a single word or short phrase.")
• Returns formatted question string

normalize_answer

def normalize_answer(answer: str) -> str

Normalizes answer for comparison:

• Converts to lowercase and strips whitespace
• Removes punctuation using regex
• Collapses multiple spaces to single space
• Returns normalized string

compute_f1_score

def compute_f1_score(prediction: str, ground_truth: str) -> float

Computes token-level F1 score:

• Tokenizes normalized prediction and ground truth
• Finds common tokens (set intersection)
• Calculates precision = |common| / |pred_tokens|
• Calculates recall = |common| / |gt_tokens|
• Returns F1 = 2 * (precision * recall) / (precision + recall)
• Returns 0.0 if no common tokens or empty inputs

compute_exact_match

def compute_exact_match(prediction: str, ground_truth: str) -> float

Computes exact match score:

• Normalizes both prediction and ground truth
• Returns 1.0 if normalized strings match exactly
• Returns 0.0 otherwise

mmsearch_plus_process_results

def mmsearch_plus_process_results(doc: Dict[str, Any], result: List[str]) -> Dict[str, Any]

Processes results with multi-reference scoring:

• Decrypts document
• Extracts first prediction (strips whitespace)
• Gets ground truth answers (handles string or list)
• Computes max F1 and exact match across all valid GT answers
• Returns dict with:
  • "f1_score": max F1 score
  • "exact_match": max exact match score

Implementation Details

• Automatic document decryption at each processing stage
• Multi-image support (variable number via num_images)
• Multiple reference answers supported
• Token-level F1 scoring with set-based comparison
• Normalization: lowercase, no punctuation, single spaces
• Graceful handling of empty predictions/answers
• Max scoring across multiple ground truth references