Implementation:Open compass VLMEvalKit Hipho Verifier
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Mathematics, Answer Verification |
Overview
Provides a high-recall math answer verification system for the Hipho benchmark, combining symbolic, numeric, and LLM-based grading approaches.
Description
This module implements comprehensive mathematical answer verification based on HuggingFace math_verify, verl, and open reasoner zero. The timeout decorator provides configurable timeout protection for verification operations. Core verification uses math_verify library's parse and verify functions with LaTeX and expression extraction configs. It includes sympy-based symbolic comparison, decimal precision handling, and fallback to LLM judge models for complex cases. The system handles LaTeX expressions, boxed answers, numeric values with units, and various mathematical notation formats through multiple parsing strategies.
Usage
Called internally by the Hipho dataset class during mathematical answer evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/hipho_verifier.py, Lines: L1-1347 - Import:
from vlmeval.dataset.utils.hipho_verifier import timeout, verify
Key Functions:
def timeout(timeout_seconds: int = 10): ...
def extract_answer(response): ...
def verify_answer(prediction, reference): ...
def math_verify_check(prediction, reference): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Predicted answer string and reference answer string in LaTeX, numeric, or natural language format |
| Outputs | Boolean indicating mathematical equivalence; optional confidence score from LLM judge |
Usage Examples
# Internal usage example
from vlmeval.dataset.utils.hipho_verifier import verify_answer
is_correct = verify_answer("\\frac{3}{4}", "0.75")