Implementation:EvolvingLMMs Lab Lmms eval Air Bench Utils

File: lmms_eval/tasks/air_bench/utils.py (279 lines)

Principle: Task Utility Functions

Overview

Utility functions for the AIR-Bench audio understanding benchmark. Supports both chat (open-ended) and foundation (multiple-choice) task types with GPT-based evaluation for chat tasks and accuracy scoring for foundation tasks.

Configuration

• GPT_EVAL_MODEL_NAME - Default: "gpt-4o-2024-11-20"
• API_TYPE - "openai" or "azure" (default: "azure")
• NUM_SECONDS_TO_SLEEP - 5 seconds between retries
• retries - 3 retry attempts

Chat Task Functions

air_bench_doc_to_audio

def air_bench_doc_to_audio(doc)

Extracts audio input from document.

Returns: List containing audio data

air_bench_doc_to_text_chat

def air_bench_doc_to_text_chat(doc, lmms_eval_specific_kwargs)

Formats question for chat tasks with pre/post prompts.

Returns: Formatted question string

get_eval

def get_eval(max_tokens: int, content: str, retries: int = retries)

Calls GPT API for evaluation with retry logic.

Parameters:

• max_tokens - Maximum response length
• content - Evaluation prompt
• retries - Retry attempts (default: 3)

Returns: Tuple of (response content, model name)

air_bench_process_results_chat

def air_bench_process_results_chat(doc, result)

Processes chat results using GPT evaluation with position bias mitigation.

Parameters:

• doc - Document with question, answer_gt, meta_info
• result - Model's response

Returns: Dictionary with "gpt_eval" containing two evaluation scores

Position Bias Mitigation: Evaluates twice with model as assistant 1 and assistant 2

Evaluation Prompt: Rates helpfulness, relevance, accuracy, comprehensiveness on 1-10 scale

air_bench_aggregate_results_chat

def air_bench_aggregate_results_chat(results)

Aggregates chat evaluation scores.

Returns: Average score divided by 2 (normalized)

Parsing: Extracts scores using regex pattern for numbers 1-10

Foundation Task Functions

air_bench_doc_to_text_foundation

def air_bench_doc_to_text_foundation(doc, lmms_eval_specific_kwargs)

Formats multiple-choice question with options A-D.

Returns: Question string with formatted choices

air_bench_doc_to_target_foundation

def air_bench_doc_to_target_foundation(doc)

Gets ground truth answer using get_gt helper.

Returns: Letter of correct choice (A/B/C/D)

air_bench_doc_to_choice_foundation

def air_bench_doc_to_choice_foundation(doc)

Extracts available choices from document.

Returns: List of choice letters (e.g., ["A", "B", "C", "D"])

parse_multi_choice_response

def parse_multi_choice_response(response, all_choices)

Parses predicted choice from model response.

Parameters:

• response - Model's text response
• all_choices - Valid choice letters

Returns: Predicted choice letter

Parsing Strategy:

1. Choices with parentheses: (A), (B)
2. Simple choices with spaces: " A ", " B "
3. Choices with periods: A., B.
4. Random choice if no match found
5. Last occurrence if multiple matches

get_gt

def get_gt(doc)

Maps answer_gt text to corresponding choice letter.

Returns: Letter (A/B/C/D) matching the ground truth answer

air_bench_process_results_foundation

def air_bench_process_results_foundation(doc, result)

Processes foundation task results with accuracy scoring.

Returns: Dictionary with "accuracy" (score and task name) and "submission" (uniq_id: pred)

air_bench_aggregate_results_foundation

def air_bench_aggregate_results_foundation(results)

Aggregates foundation results with categorical breakdown.

Returns: Overall accuracy (0-1 scale)

Logging: Outputs overall accuracy and per-task accuracy

air_bench_aggregate_results_for_submission

def air_bench_aggregate_results_for_submission(results, args)

Generates submission file for test set evaluation.

Output: JSON file with predictions

Usage Patterns

Chat Tasks:

1. Load audio input
2. Format question with meta_info
3. Get model response
4. Evaluate twice (position bias mitigation)
5. Aggregate scores

Foundation Tasks:

1. Load audio input
2. Format multiple-choice question
3. Parse model's choice
4. Calculate accuracy
5. Generate submission file

Dependencies

• json, os, random, re, time - Standard library
• numpy - Array operations
• requests - API calls
• yaml - Configuration
• loguru - Logging
• lmms_eval.tasks._task_utils.file_utils - Submission file generation