Implementation:Open compass VLMEvalKit TamperBench Utils
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Image Tampering Detection |
Overview
Provides evaluation utilities for the TamperBench image tampering detection benchmark, computing classification metrics per tamper type.
Description
This module implements get_dimension_rating for per-category accuracy computation and process_results for comprehensive classification metrics using scikit-learn. The process_results function computes accuracy, precision, recall, F1 score, confusion matrices, and ROC-AUC scores per tamper type, combining original (non-tampered) images with each tamper category for binary classification assessment. It handles prediction column generation from score/answer pairs and generates grouped metrics excluding the 'original' category from output while including it in calculations for proper binary classification evaluation.
Usage
Called internally by the TamperBench dataset class during tampering detection evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/tamperbench.py, Lines: L1-737 - Import:
from vlmeval.dataset.utils.tamperbench import get_dimension_rating, process_results
Key Functions:
def get_dimension_rating(data_path, category_type='task_type'): ...
def process_results(score_file, model_name): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Scored data file path with predictions, answers, scores, and tamper_type columns; model name string |
| Outputs | Dictionary of per-tamper-type classification metrics (accuracy, precision, recall, F1, AUC); overall accuracy breakdown |
Usage Examples
# Internal usage example
from vlmeval.dataset.utils.tamperbench import process_results
metrics = process_results("scored_results.xlsx", "model_name")