Implementation:Open compass VLMEvalKit MEGABench Derive Breakdown
| Field | Value |
|---|---|
| source | VLMEvalKit |
| domain | Vision, Evaluation, Results Aggregation, Statistics |
Overview
Derives breakdown statistics and summary scores for MEGA-Bench evaluation results across core and open task categories.
Description
This module implements `calculate_model_summary` which separates task results into core (rule-based) and open (LLM-judged) categories, then computes summary statistics including number of evaluated tasks, total samples, and macro-mean scores. It calculates an overall weighted score based on the number of tasks in each category. The module integrates with `analysis_utils` for task metadata collection and keyword-based statistical breakdowns via `derive_keyword_stats`.
Usage
Called internally by the corresponding dataset class during evaluation.
Code Reference
- Source:
vlmeval/dataset/utils/megabench/tools/derive_breakdown_results.py, Lines: L1-141 - Import:
from vlmeval.dataset.utils.megabench.tools.derive_breakdown_results import calculate_model_summary
Key Functions:
def calculate_model_summary(task_results_with_meta): ...
I/O Contract
| Direction | Description |
|---|---|
| Inputs | Dictionary of task results with metadata including 'eval_type', 'score', and 'num_query' fields |
| Outputs | Dictionary with 'core', 'open', and 'overall_score' keys containing summary statistics |
Usage Examples
from vlmeval.dataset.utils.megabench.tools.derive_breakdown_results import calculate_model_summary
summary = calculate_model_summary(task_results)
print(summary['overall_score'])
Related Pages
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment