Implementation:Open compass VLMEvalKit MMNIAH Utils
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, VQA, Contraction Handling |
Overview
Provides VQA evaluation utilities for the MM-NIAH (MultiModal Needle-In-A-Haystack) benchmark with comprehensive text normalization and contraction handling.
Description
This module implements the `VQAEval` class with extensive English contraction mappings (100+ entries) for text normalization, and `has_word` for whole-word boundary matching using regex. The contraction dictionary enables fair comparison between model outputs and ground truth by expanding common contractions. The module supports the standard VQA evaluation protocol where answers are normalized before comparison.
Usage
Called internally by the corresponding dataset class during evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/mmniah.py, Lines: L1-298 - Import:
from vlmeval.dataset.utils.mmniah import VQAEval, has_word
Key Functions:
class VQAEval:
def __init__(self): ... # Initializes contraction mappings
def has_word(sentence, word): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Prediction and ground truth answer strings for VQA comparison |
| Outputs | Boolean word-match results; normalized text for comparison |
Usage Examples
from vlmeval.dataset.utils.mmniah import has_word
found = has_word("The cat is on the mat", "cat")
# found = True
Related Pages
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment