Jump to content

Connect SuperML | Leeroopedia MCP: Equip your AI agents with best practices, code verification, and debugging knowledge. Powered by Leeroo — building Organizational Superintelligence. Contact us at founders@leeroo.com.

Implementation:EvolvingLMMs Lab Lmms eval GroundingMe YAML Config

From Leeroopedia

File: `/tmp/kapso_repo_sslb_59s/lmms_eval/tasks/groundingme/groundingme.yaml`

Principle: YAML_Task_Configuration

Overview

The GroundingMe YAML configuration defines a comprehensive visual grounding evaluation task with 48 distinct metrics spanning multiple evaluation dimensions including IoU scores, accuracy thresholds, spatial reasoning, discriminative capabilities, and rejection accuracy.

Configuration Structure

Basic Task Setup

task: "groundingme"
output_type: generate_until
test_split: test
dataset_path: lirang04/GroundingME

Document Processing

doc_to_visual: !function utils.groundingme_doc_to_visual
doc_to_text: !function utils.groundingme_doc_to_text
doc_to_target: ""

Custom utility functions handle visual and text extraction, with an empty target string indicating no direct target field.

Generation Configuration

generation_kwargs:
  max_new_tokens: 20480
  temperature: 0.0
  do_sample: false

Notably high token limit (20,480) for potentially generating detailed grounding outputs, with deterministic generation.

Result Processing

process_results: !function utils.groundingme_process_result

Comprehensive Metric List

The configuration defines 48 metrics organized into several categories:

Core Metrics

- metric: groundingme_IoU
  aggregation: !function utils.groundingme_iou
  higher_is_better: true
- metric: groundingme_ACC@0.5
  aggregation: !function utils.groundingme_acc05
  higher_is_better: true
- metric: groundingme_ACC@0.75
  aggregation: !function utils.groundingme_acc075
  higher_is_better: true
- metric: groundingme_ACC@0.9
  aggregation: !function utils.groundingme_acc09
  higher_is_better: true
- metric: groundingme_Center_ACC
  aggregation: !function utils.groundingme_center_acc
  higher_is_better: true
- metric: groundingme_MACC
  aggregation: !function utils.groundingme_macc
  higher_is_better: true

Core grounding metrics including IoU, accuracy at multiple IoU thresholds (0.5, 0.75, 0.9), center accuracy, and mean accuracy.

Discriminative Task Metrics

- metric: groundingme_Discriminative_ACC@0.5
  aggregation: !function utils.groundingme_discriminative_acc05
  higher_is_better: true
- metric: groundingme_Discriminative_ACC@0.75
  aggregation: !function utils.groundingme_discriminative_acc075
  higher_is_better: true
- metric: groundingme_Discriminative_ACC@0.9
  aggregation: !function utils.groundingme_discriminative_acc09
  higher_is_better: true
- metric: groundingme_Discriminative_MACC
  aggregation: !function utils.groundingme_discriminative_macc
  higher_is_better: true

Spatial Reasoning Metrics

- metric: groundingme_Spatial_ACC@0.5
  aggregation: !function utils.groundingme_spatial_acc05
  higher_is_better: true
- metric: groundingme_Spatial_ACC@0.75
  aggregation: !function utils.groundingme_spatial_acc075
  higher_is_better: true
- metric: groundingme_Spatial_ACC@0.9
  aggregation: !function utils.groundingme_spatial_acc09
  higher_is_better: true
- metric: groundingme_Spatial_MACC
  aggregation: !function utils.groundingme_spatial_macc
  higher_is_better: true

Limited Context Metrics

- metric: groundingme_Limited_ACC@0.5
  aggregation: !function utils.groundingme_limited_acc05
  higher_is_better: true
- metric: groundingme_Limited_ACC@0.75
  aggregation: !function utils.groundingme_limited_acc075
  higher_is_better: true
- metric: groundingme_Limited_ACC@0.9
  aggregation: !function utils.groundingme_limited_acc09
  higher_is_better: true
- metric: groundingme_Limited_MACC
  aggregation: !function utils.groundingme_limited_macc
  higher_is_better: true

Rejection Accuracy

- metric: groundingme_Rejection_ACC
  aggregation: !function utils.groundingme_rejection_acc
  higher_is_better: true

Discriminative Attribute-Specific Metrics

The configuration includes metrics for four discriminative categories:

Appearance

- metric: groundingme_D_Appearance_ACC@0.5
  aggregation: !function utils.groundingme_d_appearance_acc05
  higher_is_better: true
- metric: groundingme_D_Appearance_ACC@0.75
  aggregation: !function utils.groundingme_d_appearance_acc075
  higher_is_better: true
- metric: groundingme_D_Appearance_ACC@0.9
  aggregation: !function utils.groundingme_d_appearance_acc09
  higher_is_better: true
- metric: groundingme_D_Appearance_MACC
  aggregation: !function utils.groundingme_d_appearance_macc
  higher_is_better: true

Component

- metric: groundingme_D_Component_ACC@0.5
  aggregation: !function utils.groundingme_d_component_acc05
  higher_is_better: true
- metric: groundingme_D_Component_ACC@0.75
  aggregation: !function utils.groundingme_d_component_acc075
  higher_is_better: true
- metric: groundingme_D_Component_ACC@0.9
  aggregation: !function utils.groundingme_d_component_acc09
  higher_is_better: true
- metric: groundingme_D_Component_MACC
  aggregation: !function utils.groundingme_d_component_macc
  higher_is_better: true

Text

- metric: groundingme_D_Text_ACC@0.5
  aggregation: !function utils.groundingme_d_text_acc05
  higher_is_better: true
- metric: groundingme_D_Text_ACC@0.75
  aggregation: !function utils.groundingme_d_text_acc075
  higher_is_better: true
- metric: groundingme_D_Text_ACC@0.9
  aggregation: !function utils.groundingme_d_text_acc09
  higher_is_better: true
- metric: groundingme_D_Text_MACC
  aggregation: !function utils.groundingme_d_text_macc
  higher_is_better: true

State

- metric: groundingme_D_State_ACC@0.5
  aggregation: !function utils.groundingme_d_state_acc05
  higher_is_better: true
- metric: groundingme_D_State_ACC@0.75
  aggregation: !function utils.groundingme_d_state_acc075
  higher_is_better: true
- metric: groundingme_D_State_ACC@0.9
  aggregation: !function utils.groundingme_d_state_acc09
  higher_is_better: true
- metric: groundingme_D_State_MACC
  aggregation: !function utils.groundingme_d_state_macc
  higher_is_better: true

Reasoning Task Metrics

Relationship

- metric: groundingme_Relationship_ACC@0.5
  aggregation: !function utils.groundingme_relationship_acc05
  higher_is_better: true
- metric: groundingme_Relationship_ACC@0.75
  aggregation: !function utils.groundingme_relationship_acc075
  higher_is_better: true
- metric: groundingme_Relationship_ACC@0.9
  aggregation: !function utils.groundingme_relationship_acc09
  higher_is_better: true
- metric: groundingme_Relationship_MACC
  aggregation: !function utils.groundingme_relationship_macc
  higher_is_better: true

Counting

- metric: groundingme_Counting_ACC@0.5
  aggregation: !function utils.groundingme_counting_acc05
  higher_is_better: true
- metric: groundingme_Counting_ACC@0.75
  aggregation: !function utils.groundingme_counting_acc075
  higher_is_better: true
- metric: groundingme_Counting_ACC@0.9
  aggregation: !function utils.groundingme_counting_acc09
  higher_is_better: true
- metric: groundingme_Counting_MACC
  aggregation: !function utils.groundingme_counting_macc
  higher_is_better: true

Occlusion

- metric: groundingme_Occlusion_ACC@0.5
  aggregation: !function utils.groundingme_occlusion_acc05
  higher_is_better: true
- metric: groundingme_Occlusion_ACC@0.75
  aggregation: !function utils.groundingme_occlusion_acc075
  higher_is_better: true
- metric: groundingme_Occlusion_ACC@0.9
  aggregation: !function utils.groundingme_occlusion_acc09
  higher_is_better: true
- metric: groundingme_Occlusion_MACC
  aggregation: !function utils.groundingme_occlusion_macc
  higher_is_better: true

Small Objects

- metric: groundingme_Small_ACC@0.5
  aggregation: !function utils.groundingme_small_acc05
  higher_is_better: true
- metric: groundingme_Small_ACC@0.75
  aggregation: !function utils.groundingme_small_acc075
  higher_is_better: true
- metric: groundingme_Small_ACC@0.9
  aggregation: !function utils.groundingme_small_acc09
  higher_is_better: true
- metric: groundingme_Small_MACC
  aggregation: !function utils.groundingme_small_macc
  higher_is_better: true

Rejection Attribute-Specific Metrics

- metric: groundingme_R_Appearance_ACC
  aggregation: !function utils.groundingme_r_appearance_acc
  higher_is_better: true
- metric: groundingme_R_Component_ACC
  aggregation: !function utils.groundingme_r_component_acc
  higher_is_better: true
- metric: groundingme_R_Text_ACC
  aggregation: !function utils.groundingme_r_text_acc
  higher_is_better: true
- metric: groundingme_R_State_ACC
  aggregation: !function utils.groundingme_r_state_acc
  higher_is_better: true

Metadata

metadata:
  version: '0.0'

Design Patterns

Multi-Threshold Evaluation

The configuration evaluates grounding accuracy at three IoU thresholds (0.5, 0.75, 0.9), providing a comprehensive view of model precision across different strictness levels.

Hierarchical Task Breakdown

Metrics are organized hierarchically: 1. Global metrics: Overall IoU and accuracy 2. Task-type metrics: Discriminative, spatial, limited, rejection 3. Attribute-specific metrics: Appearance, component, text, state 4. Reasoning-specific metrics: Relationship, counting, occlusion, small objects

Dedicated Aggregation Functions

Each metric has a dedicated aggregation function, allowing for specialized processing logic while maintaining a consistent naming convention (`groundingme_<category>_<metric>`).

Fine-Grained Analysis

The extensive metric breakdown enables detailed performance analysis across multiple dimensions, helping identify specific weaknesses in visual grounding capabilities.

Related Components

  • Utility functions: `lmms_eval/tasks/groundingme/utils.py`
  • Dataset: `lirang04/GroundingME` on Hugging Face Hub
  • Principle: YAML_Task_Configuration

Page Connections

Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment
Retrieved from "https://leeroopedia.com/index.php?title=Implementation:EvolvingLMMs_Lab_Lmms_eval_GroundingMe_YAML_Config&oldid=6276"