Implementation:Open compass VLMEvalKit PhyX Utils
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Physics, Answer Verification |
Overview
Provides physics answer evaluation utilities for the PhyX benchmark, using in-context examples and LLM-as-judge for answer equivalence with unit handling.
Description
This module implements physics answer grading using get_ICE which provides five in-context examples demonstrating how to judge equivalence between predicted and ground-truth physics answers. The examples cover unit conversions (kN to N), approximate numerical matching, mass calculations, and distance estimations with mathematical expressions. The evaluation system handles physics-specific challenges such as unit conversion tolerance, significant figure matching, and boxed LaTeX answer extraction. It leverages LLM-based judging with binary (0/1) judgement outputs.
Usage
Called internally by the PhyX dataset class during physics answer evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/phyx.py, Lines: L1-378 - Import:
from vlmeval.dataset.utils.phyx import get_ICE
Key Functions:
def get_ICE(): ...
def evaluate_physics_answer(prediction, reference, model): ...
def extract_boxed_answer(response): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Predicted physics answer string with optional units; ground-truth answer string; LLM judge model |
| Outputs | Binary judgement (0 or 1) indicating answer correctness; extracted answer strings from boxed LaTeX |
Usage Examples
# Internal usage example
from vlmeval.dataset.utils.phyx import get_ICE
examples = get_ICE() # Returns 5 in-context examples for physics answer grading