Implementation:EvolvingLMMs Lab Lmms eval VisualPuzzles Utils

File: lmms_eval/tasks/VisualPuzzles/utils.py (116 lines)

Principle: Task Utility Functions

Overview

Utility functions for the VisualPuzzles benchmark evaluation task. Provides document-to-visual/text conversions, answer parsing with multiple regex patterns, and result processing for multiple-choice visual reasoning questions.

Key Functions

VisualPuzzles_doc_to_visual

def VisualPuzzles_doc_to_visual(doc)

Extracts the image from a document for visual input.

Parameters:

• doc - Document instance containing image data

Returns: List containing the document's image

VisualPuzzles_doc_to_text

def VisualPuzzles_doc_to_text(doc, lmms_eval_specific_kwargs)

Formats question text with options and prompt instructions.

Parameters:

• doc - Document with question and options
• lmms_eval_specific_kwargs - Contains prompt type selection

Returns: Formatted question string with options (A-D) and instructions

Prompt Types:

• MULTI_CHOICE_DIRECT_PROMPT - Direct answer instruction
• COT_PROMPT - Chain-of-thought reasoning with answer format

parse_response

def parse_response(response, all_choices, index2ans)

Extracts answer choice from model response using multiple regex patterns.

Parameters:

• response - Model's text response
• all_choices - Valid answer choices (e.g., ["A", "B", "C", "D"])
• index2ans - Optional mapping from choice letters to answer text

Returns: Extracted answer letter or random choice if parsing fails

Parsing Strategy: Tries patterns in order:

1. "Answer: (A)" format
2. "Answer: A" format (excluding "Final Answer:")
3. Parentheses: (A), (B), etc.
4. Letter with parenthesis: A), B), etc.
5. Braces: {A}, {B}, etc.
6. Dollar signs: $A$, $B$, etc.
7. Letter with period: A., B., etc.
8. Standalone letters (for short responses)
9. Searches for answer text from index2ans mapping
10. Returns random choice if all patterns fail

VisualPuzzles_process_result

def VisualPuzzles_process_result(doc, results)

Processes model results and compares with ground truth.

Parameters:

• doc - Document with answer key and options
• results - Model's response list

Returns: Dictionary with "exact_match" score (1.0 or 0.0)

Constants

• MULTI_CHOICE_DIRECT_PROMPT - "Answer the question with the option's letter from the given choices directly."
• COT_PROMPT - Chain-of-thought instruction with specific answer format requirement
• PROMPTS - Dictionary mapping prompt names to prompt strings

Usage Pattern

The module follows the standard task utility pattern:

1. Convert document to visual input (image)
2. Format question with options and prompt
3. Get model response
4. Parse response using robust regex patterns
5. Calculate exact match accuracy

Dependencies

• random - For fallback random choice selection
• re - For regex-based answer extraction