Implementation:EvolvingLMMs Lab Lmms eval MultiDocVQA Utils

Task utility functions for the MultiDocVQA benchmark, which evaluates visual question answering across multiple document images.

Location

/tmp/kapso_repo_sslb_59s/lmms_eval/tasks/multidocvqa/utils.py

Overview

Provides document processing, result handling, and ANLS (Average Normalized Levenshtein Similarity) evaluation for MultiDocVQA tasks. The module supports both validation (with accuracy/ANLS metrics) and test set submission.

Core Functions

Document Processing

multidocvqa_doc_to_text(doc, lmms_eval_specific_kwargs)

Constructs prompt from question with pre/post prompts

Parameters:

• doc - Document with "question" key
• lmms_eval_specific_kwargs - Dict with pre_prompt and post_prompt

Returns: Formatted prompt string

multidocvqa_doc_to_visual(doc)

Extracts up to 20 images from document

Parameters: doc - Document with image_1 through image_20 keys

Returns: List of RGB-converted PIL Images (non-null images only)

Result Processing

multidocvqa_process_results(doc, results)

Packages prediction with ground truth answers for validation metrics

Parameters:

• doc - Document with questionId and answers (as string)
• results - Model prediction list

Returns: Dictionary with anls and accuracy entries containing:

• questionId (int)
• answer (list, parsed from string)
• pred_answer (string)

multidocvqa_process_test_results_for_submission(doc, results)

Formats test set predictions for submission

Parameters: doc, results

Returns: Dictionary with submission entry containing questionId, answer, answer_page

Aggregation Functions

multidocvqa_aggregate_results_anls(results)

Computes average ANLS score across all results

Parameters: results - List of result dictionaries

Process:

1. Restructures results by key
2. Initializes Evaluator (case-insensitive)
3. Computes ANLS metrics

Returns: Mean ANLS score (float)

multidocvqa_aggregate_results_accuracy(results)

Computes average exact match accuracy

Parameters: results

Process: Same as ANLS but returns accuracy metric

Returns: Mean accuracy (float)

multidocvqa_test_aggregate_results_for_submission(results, args)

Saves test results to JSON file for submission

Parameters:

• results - Test results list
• args - Arguments for file path generation

Side Effect: Writes multidocvqa_test_for_submission.json

Evaluator Class

Evaluator

Helper class for computing accuracy and ANLS metrics.

Initialization

Evaluator(case_sensitive=False)

• case_sensitive (bool): Whether to preserve case in comparisons
• anls_threshold: Fixed at 0.5
• get_edit_distance: Uses Levenshtein distance function

Methods

get_metrics(gt_answers, preds)

Computes accuracy and ANLS for batches

Parameters:

• gt_answers - List of answer lists
• preds - List of prediction strings

Returns: Dictionary with accuracy and anls lists

_preprocess_str(string)

Lowercases (if configured) and strips whitespace

Returns: Preprocessed string

_calculate_accuracy(gt, pred)

Returns 1 if prediction exactly matches any ground truth, else 0

Special case: Returns 0 if prediction is "none"

_calculate_anls(gt, pred)

Computes ANLS score using normalized Levenshtein distance

Process:

1. Returns 0 for empty or "none" predictions
2. Computes similarity for each ground truth
3. Takes maximum similarity
4. Returns similarity if ≥ 0.5, else 0

Returns: ANLS score (float)

Dependencies

• ast, json
• loguru.logger as eval_logger
• lmms_eval.api.metrics.levenshtein_distance
• lmms_eval.tasks._task_utils.file_utils.generate_submission_file

Usage Example

The module includes a main block for testing:

multidocvqa_aggregate_results_anls([
    {"questionId": 1, "answer": ["answer"], "pred_answer": "pred_answer"},
    {"questionId": 2, "answer": ["nswer"], "pred_answer": "nswer"}
])

Related

• Task_Utility_Functions - General task utility pattern
• ANLS_Metric - Average Normalized Levenshtein Similarity details