Implementation:Open compass VLMEvalKit BMMR Grade
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Mathematics, Answer Grading |
Overview
Provides mathematical answer equivalence checking for the BMMR (Bilingual Multimodal Mathematical Reasoning) benchmark evaluation.
Description
This module implements comprehensive math answer grading logic adapted from Hendrycks' MATH benchmark, Microsoft ToRA, and OpenAI PRM800K. Key functions include is_digit for numeric detection, is_equiv for symbolic math equivalence checking using SymPy, and LaTeX expression parsing via parse_latex. The grading system handles numeric comparisons with tolerance, symbolic simplification, set/interval notation, matrix comparisons, and various mathematical expression formats to determine answer correctness.
Usage
Called internally by the BMMR dataset class during mathematical answer evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/bmmr_grade.py, Lines: L1-470 - Import:
from vlmeval.dataset.utils.bmmr_grade import is_equiv, is_digit
Key Functions:
def is_digit(s): ...
def is_equiv(str1, str2, verbose=False): ...
def numeric_equal(prediction, reference): ...
def symbolic_equal(a, b): ...
def math_equal(prediction, reference): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Two mathematical answer strings (prediction and reference) in LaTeX, numeric, or symbolic format |
| Outputs | Boolean indicating whether the two answers are mathematically equivalent |
Usage Examples
# Internal usage example
from vlmeval.dataset.utils.bmmr_grade import is_equiv
result = is_equiv("\\frac{1}{2}", "0.5") # Returns True
Related Pages
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment