Implementation:Open compass VLMEvalKit VLMsAreBiased Utils
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Bias Detection, Fairness |
Overview
Provides bias detection and accuracy evaluation utilities for the VLMs-Are-Biased benchmark measuring model prediction bias against expected bias patterns.
Description
This module implements `vlms_are_biased_process_results` which processes evaluation results by comparing predictions against both ground truth and expected bias patterns. It normalizes predictions (lowering, stripping braces) and uses both text and number-only matching for robustness. The `vlms_are_biased_aggregate_by_topic` function aggregates results by topic, computing overall accuracy, macro accuracy (averaged across topics), and bias ratio (proportion of incorrect answers that match the expected bias).
Usage
Called internally by the corresponding dataset class during evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/vlmsarebiased.py, Lines: L1-93 - Import:
from vlmeval.dataset.utils.vlmsarebiased import vlms_are_biased_process_results, vlms_are_biased_aggregate_by_topic
Key Functions:
def vlms_are_biased_process_results(org_file): ...
def vlms_are_biased_aggregate_by_topic(detail_result): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Path to results file with 'prediction', 'ground_truth', 'expected_bias', and 'topic' fields |
| Outputs | Enriched results with is_correct/matches_bias flags; aggregate dict with overall_acc, macro_acc, and bias_ratio |
Usage Examples
from vlmeval.dataset.utils.vlmsarebiased import vlms_are_biased_process_results
results = vlms_are_biased_process_results("results.tsv")