Implementation:EvolvingLMMs Lab Lmms eval BabyVision Utils

File: lmms_eval/tasks/babyvision/utils.py (245 lines)

Principle: Task Utility Functions

Overview

Utility functions for the BabyVision benchmark evaluation. Uses LLM-as-Judge API to evaluate visual question answering with support for multiple-choice and open-ended questions. Includes special handling for models with reasoning tokens (e.g., MiMo-VL's <think> format).

Configuration

Environment variables:

• BABYVISION_API_KEY - Required OpenAI API key
• BABYVISION_BASE_URL - Optional custom API endpoint
• BABYVISION_MODEL_NAME - Judge model (default: "gpt-4o")

Key Functions

_get_openai_client

def _get_openai_client() -> Optional["OpenAI"]

Creates OpenAI client instance for LLM judge API calls.

Returns: OpenAI client or None if API key not found

parse_bool_response

def parse_bool_response(response_text: str) -> Optional[bool]

Parses boolean correctness from LLM judge response.

Parameters:

• response_text - Judge's response string

Returns: True if "true" in response, False if "false", None if unparseable

extract_boxed_answer

def extract_boxed_answer(text: str) -> Optional[str]

Extracts answer from model response, handling multiple formats.

Parameters:

• text - Model's response text

Returns: Extracted answer string or None

Extraction Strategy:

1. Priority: Content after </think> tag (for reasoning models)
2. Primary: \boxed{} LaTeX-style format
3. Fallback: "answer is X" or "Answer: X" patterns
4. Regex: Handles nested braces and various answer formats

Supported Formats:

• \boxed{answer} - LaTeX style
• answer is X / Answer: X
• final answer: X / Final Answer: X

normalize_answer

def normalize_answer(answer: str) -> str

Normalizes answer for comparison.

Operations:

• Converts to lowercase
• Removes extra whitespace
• Strips trailing punctuation

babyvision_doc_to_visual

def babyvision_doc_to_visual(doc)

Extracts and converts image to RGB format.

Returns: List with single RGB PIL Image

babyvision_doc_to_text

def babyvision_doc_to_text(doc, lmms_eval_specific_kwargs=None)

Formats question text with optional choices for multiple-choice questions.

Parameters:

• doc - Document with question, ansType, and optional options
• lmms_eval_specific_kwargs - Contains pre_prompt and post_prompt

Returns: Formatted question string

Choice Format: For multiple-choice (ansType=="choice"):

{question}
Choices:
(A) option1
(B) option2
...

babyvision_doc_to_target

def babyvision_doc_to_target(doc)

Extracts ground truth answer from document.

Returns:

• Letter (A, B, C, ...) for choice questions (from choiceAns index)
• String answer for blank questions (from blankAns)
• Empty string if no answer

babyvision_process_results

def babyvision_process_results(doc, results, **kwargs)

Processes results using LLM-as-Judge evaluation.

Parameters:

• doc - Document with type, subtype, question, answer
• results - Model's response list

Returns: Dictionary with "babyvision_overall_accuracy" containing:

• task_type, subtype - For categorical breakdown
• correct - Boolean correctness from judge
• parsed_pred, groundtruth - Extracted and ground truth answers
• error - Error message if API call fails

Evaluation Process:

1. Extract answer using extract_boxed_answer
2. Build judge prompt from question, groundtruth, parsed prediction
3. Call GPT judge API (max 10 tokens, temperature 0.0)
4. Parse boolean correctness from response
5. Return detailed result dictionary

Error Handling:

• Returns correct=False with error message on API failures
• Logs errors for debugging

babyvision_aggregate_results

def babyvision_aggregate_results(results)

Aggregates results with categorical breakdowns.

Parameters:

• results - List of result dictionaries from process_results

Returns: Float accuracy (0.0-1.0)

Logging Output:

• Overall accuracy with counts
• Error count (if any)
• Accuracy by task_type/subtype

Categorical Tracking: Groups by "{task_type}/{subtype}" for detailed analysis

Usage Pattern

1. Convert image to RGB format
2. Format question (with choices if multiple-choice)
3. Get model response
4. Extract answer from response (handling reasoning tokens)
5. Build judge prompt and call LLM judge API
6. Parse boolean correctness
7. Aggregate with categorical breakdowns

Dependencies

• os, re - Standard library
• collections.defaultdict - Categorical counting
• typing.Optional - Type hints
• loguru - Logging
• lmms_eval.tasks.babyvision.prompt - Judge prompt builder
• openai.OpenAI - API client (imported lazily)