Implementation:Open compass VLMEvalKit OCRBench V2 Eval
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, OCR, Document Understanding |
Overview
Implements the main evaluation pipeline for OCRBench v2, orchestrating multiple OCR-specific metrics across diverse document understanding tasks.
Description
This module provides process_predictions as the central evaluation function, dispatching to task-specific evaluation metrics based on the prediction type. It integrates TEDS metric for table structure recognition, VQA evaluation for text recognition and reasoning tasks, IoU-based evaluation for position-aware tasks, page OCR metrics, and spotting evaluation for text detection. The is_nan_value and get_value_or_zero utilities handle missing values, while calculate_average aggregates scores across task categories. The module supports both English and Chinese evaluation variants across categories like APP agents, ASCII art, math QA, science QA, and document classification.
Usage
Called internally by the OCRBench v2 dataset class during comprehensive OCR evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/ocrbrnch_v2_eval.py, Lines: L1-441 - Import:
from vlmeval.dataset.utils.ocrbrnch_v2_eval import process_predictions, calculate_average
Key Functions:
def is_nan_value(value): ...
def get_value_or_zero(value): ...
def calculate_average(scores_dict): ...
def process_predictions(predict_file): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | List of prediction dictionaries with type, answers, predict, and eval fields |
| Outputs | List of scored result dictionaries with per-item scores; averaged scores per category |
Usage Examples
# Internal usage example
from vlmeval.dataset.utils.ocrbrnch_v2_eval import process_predictions
results = process_predictions(prediction_list)