Implementation:EvolvingLMMs Lab Lmms eval MMUPD Utils

Task utility functions for the MMUPD (Multi-Modal Understanding and Perception Disorder) benchmark, which evaluates models on their robustness to various types of perturbations (AAD, IASD, IVQD).

Location

/tmp/kapso_repo_sslb_59s/lmms_eval/tasks/mmupd/utils.py

Overview

Provides document processing, result handling, and GPT-based evaluation for MMUPD benchmark tasks. The module loads configuration from YAML, interfaces with OpenAI or Azure APIs for GPT-based evaluation, and computes dual accuracy metrics (standard and perturbed).

Key Components

Configuration

• Loads mmupd.yaml to extract system prompt
• Configures GPT evaluation model (default: gpt-4o-2024-11-20)
• Supports OpenAI and Azure API types via environment variables

Core Functions

Document Processing

mmupd_doc_to_visual(doc)

Converts base64-encoded image from document to PIL Image

Parameters: doc - Document dictionary with "image" key

Returns: List containing single PIL Image object

mmupd_doc_to_text(doc, lmms_eval_specific_kwargs=None)

Constructs query prompt with question, hint, and options

Parameters:

• doc - Document with question, hint, answer, options
• lmms_eval_specific_kwargs - Optional kwargs with post_prompt

Returns: Formatted query string

Result Processing

mmupd_process_results(doc, results)

Structures model response with metadata for GPT evaluation

Parameters:

• doc - Original document
• results - Model response list

Returns: Dictionary with gpt_eval_score and submission entries

Evaluation Functions

Each perturbation type (AAD, IASD, IVQD) has three question variants:

AAD (Attribute Ambiguity Disorder)

• mmaad_base(results, args)
• mmaad_option(results, args)
• mmaad_instruction(results, args)

IASD (Instruction Ambiguity and Semantic Disorder)

• mmiasd_base(results, args)
• mmiasd_option(results, args)
• mmiasd_instruction(results, args)

IVQD (Image-Visual Question Disorder)

• mmivqd_base(results, args)
• mmivqd_option(results, args)
• mmivqd_instruction(results, args)

All functions call mmupd_results_eval() with specific type and question type parameters.

Main Evaluation Logic

mmupd_results_eval(results, args, upd_type, question_type)

Computes dual accuracy metrics and saves results

Parameters:

• results - Collected evaluation results
• args - Arguments for file generation
• upd_type - One of "aad", "iasd", "ivqd"
• question_type - One of "base", "option", "inst"

Process:

1. Evaluates using standard method
2. Evaluates using perturbation-specific method
3. Calculates dual accuracy (both correct)
4. Saves JSON summary and Excel details

Returns: Dual accuracy percentage (0-100)

Global Evaluator

• mmupd_evaluator: Instance of MMUPD_Evaluator class
• Handles option prompt creation and OpenAI API interaction
• Configured with system prompt, API credentials, and model version

Dependencies

• base64, json, os, pathlib.Path
• pandas, yaml, PIL.Image
• lmms_eval.tasks._task_utils.file_utils.generate_submission_file
• lmms_eval.tasks.mmupd.mmupd_evals.MMUPD_Evaluator

Output Files

Each evaluation generates three files:

• mmupd_{upd_type}_{question_type}_dual_results.json - Accuracy summary
• mmupd_{upd_type}_{question_type}_dual_results_detail.xlsx - Detailed Excel
• mmupd_{upd_type}_{question_type}_dual_results_detail.json - Detailed JSON for leaderboard

Related

• MMUPD_Evaluator - Core evaluation class
• Task_Utility_Functions - General task utility pattern