Implementation:Open compass VLMEvalKit OmniDocBench

Field	Value
source	VLMEvalKit
domain	Vision, Benchmarking, Document Understanding, OCR

Overview

Benchmark dataset implementation for OmniDocBench document understanding and PDF-to-Markdown conversion evaluation in VLMEvalKit.

Description

OmniDocBench inherits from ImageBaseDataset and implements the OmniDocBench benchmark for evaluating document image understanding, including text recognition, formula conversion, and table parsing. The TYPE field is set to 'QA'. It uses a detailed system prompt instructing models to convert PDF images to Markdown format with LaTeX formulas and HTML tables. The file also defines end2end_evaluator and table_evalutor classes for comprehensive document evaluation.

Usage

Registered in vlmeval/dataset/__init__.py and invoked through build_dataset() by benchmark name.

Code Reference

Source: vlmeval/dataset/OmniDocBench/omnidocbench.py, Lines: L1-551
Import: from vlmeval.dataset.OmniDocBench.omnidocbench import OmniDocBench

Signature:

class OmniDocBench(ImageBaseDataset):
    MODALITY = 'IMAGE'
    TYPE = 'QA'
    DATASET_URL = {...}
    DATASET_MD5 = {...}
    ...

I/O Contract

Direction	Description
Inputs	TSV dataset file with document images and parsing tasks
Outputs	Evaluation results DataFrame with TEDS, BLEU, edit distance scores

Usage Examples

from vlmeval.dataset import build_dataset
dataset = build_dataset('OmniDocBench')

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment