Implementation:Open compass VLMEvalKit Omni3DBench Utils
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, 3D Understanding, Multi-type |
Overview
Provides evaluation utilities for the Omni3DBench benchmark with multi-type answer assessment including numerical, yes/no, and multi-choice questions with MRA thresholds.
Description
This module implements `extract_answer` for parsing answers from `<ans>` tags, and `Omni3DBench_acc` for computing accuracy across different answer types: integer counting (exact match), yes/no (including true/false synonyms), multi-choice (string matching), and floating-point (using MRA - Mean Relative Accuracy with 10 thresholds from 0.05 to 0.5). The prompt template `OMNI3DBENCH_PROMPT` instructs models to provide concise answers in `<ans>` tags.
Usage
Called internally by the corresponding dataset class during evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/omni3dbench.py, Lines: L1-99 - Import:
from vlmeval.dataset.utils.omni3dbench import Omni3DBench_acc, extract_answer
Key Functions:
def extract_answer(prediction): ...
def Omni3DBench_acc(data): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | DataFrame with 'answer_type', 'answer', and 'prediction' columns |
| Outputs | Dictionary with per-type accuracy scores and MRA-based metrics |
Usage Examples
from vlmeval.dataset.utils.omni3dbench import Omni3DBench_acc
results = Omni3DBench_acc(evaluation_data)
Related Pages
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment