Implementation:Open compass VLMEvalKit MEGABench Metric Type
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Benchmark Framework, Metric Types |
Overview
Defines the comprehensive enumeration of all metric types used in the MEGA-Bench evaluation framework through the MetricType enum.
Description
The `MetricType` enum class defines over 50 metric types organized across string matching (exact, simple, case-insensitive, near), numerical comparison (relative diff, RMSE), set operations (equality, Jaccard, precision), sequence operations (equality, accuracy, prefix ratio), spatial metrics (bounding box IoU, point distance, geo proximity), code evaluation (program judge), NLI, BLEU/GLEU, LaTeX expression equality, symbolic planning, and VLM-as-judge metrics. Each enum value maps to a string identifier used for metric dispatch.
Usage
Called internally by the corresponding dataset class during evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/megabench/metric_type.py, Lines: L1-259 - Import:
from vlmeval.dataset.utils.megabench.metric_type import MetricType
Key Functions:
class MetricType(Enum):
EXACT_STR_MATCH = "exact_str_match"
JACCARD_INDEX = "jaccard_index"
LATEX_EXPR_EQUALITY = "latex_expr_equality"
VLM_AS_JUDGE = "gpt_4o_as_judge"
# ... 50+ metric types
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Metric type string identifiers from task configurations |
| Outputs | MetricType enum values used for dispatching to appropriate scoring functions |
Usage Examples
from vlmeval.dataset.utils.megabench.metric_type import MetricType
metric = MetricType.EXACT_STR_MATCH
print(metric.value) # "exact_str_match"