Implementation:EvolvingLMMs Lab Lmms eval GEdit Bench YAML Config

File: `/tmp/kapso_repo_sslb_59s/lmms_eval/tasks/gedit_bench/gedit_bench.yaml`

Overview

The GEdit-Bench YAML configuration defines a task for evaluating image editing capabilities across multiple languages (English and Chinese) and subset categories (fullset and intersection). It provides a comprehensive metric breakdown with 12 separate metrics tracking semantics, quality, and overall scores.

Configuration Structure

Dataset Configuration

dataset_path: stepfun-ai/GEdit-Bench
dataset_kwargs:
  token: True
task: "gedit_bench"
test_split: train
output_type: generate_until

The dataset is loaded from Hugging Face with authentication token support, using the train split for evaluation with generative output.

Document Processing

doc_to_visual: !function utils.gedit_bench_doc_to_visual
doc_to_text: !function utils.gedit_bench_doc_to_text
doc_to_target: "instruction"

Custom utility functions handle visual and text extraction, with the target field pointing to the "instruction" key.

Generation Parameters

generation_kwargs:
  max_new_tokens: 512
  temperature: 0
  top_p: 1.0
  num_beams: 1
  do_sample: false

Deterministic generation with greedy decoding (temperature=0, no sampling, single beam).

Result Processing

process_results: !function utils.gedit_bench_process_results

Custom processing function prepares results for the metric breakdown.

Metric Breakdown

The configuration defines a hierarchical metric structure across three dimensions:

Overall Metrics

- metric: gedit_bench_semantics_score
  aggregation: !function utils.gedit_bench_aggregate_results
  higher_is_better: true
- metric: gedit_bench_quality_score
  aggregation: !function utils.gedit_bench_aggregate_results
  higher_is_better: true
- metric: gedit_bench_overall_score
  aggregation: !function utils.gedit_bench_aggregate_results
  higher_is_better: true

Global averages across all samples for three scoring dimensions.

Language-Specific Metrics

The configuration breaks down metrics by language (English and Chinese) and subset type:

English - Fullset

- metric: gedit_bench_en_fullset_semantics
  aggregation: !function utils.gedit_bench_aggregate_en_fullset_semantics
  higher_is_better: true
- metric: gedit_bench_en_fullset_quality
  aggregation: !function utils.gedit_bench_aggregate_en_fullset_quality
  higher_is_better: true
- metric: gedit_bench_en_fullset_overall
  aggregation: !function utils.gedit_bench_aggregate_en_fullset_overall
  higher_is_better: true

English - Intersection

- metric: gedit_bench_en_intersection_semantics
  aggregation: !function utils.gedit_bench_aggregate_en_intersection_semantics
  higher_is_better: true
- metric: gedit_bench_en_intersection_quality
  aggregation: !function utils.gedit_bench_aggregate_en_intersection_quality
  higher_is_better: true
- metric: gedit_bench_en_intersection_overall
  aggregation: !function utils.gedit_bench_aggregate_en_intersection_overall
  higher_is_better: true

Chinese - Fullset

- metric: gedit_bench_cn_fullset_semantics
  aggregation: !function utils.gedit_bench_aggregate_cn_fullset_semantics
  higher_is_better: true
- metric: gedit_bench_cn_fullset_quality
  aggregation: !function utils.gedit_bench_aggregate_cn_fullset_quality
  higher_is_better: true
- metric: gedit_bench_cn_fullset_overall
  aggregation: !function utils.gedit_bench_aggregate_cn_fullset_overall
  higher_is_better: true

Chinese - Intersection

- metric: gedit_bench_cn_intersection_semantics
  aggregation: !function utils.gedit_bench_aggregate_cn_intersection_semantics
  higher_is_better: true
- metric: gedit_bench_cn_intersection_quality
  aggregation: !function utils.gedit_bench_aggregate_cn_intersection_quality
  higher_is_better: true
- metric: gedit_bench_cn_intersection_overall
  aggregation: !function utils.gedit_bench_aggregate_cn_intersection_overall
  higher_is_better: true

Prompt Configuration

lmms_eval_specific_kwargs:
  default:
    pre_prompt: ""
    post_prompt: ""

Empty pre/post prompts allow the task utilities full control over prompt construction.

Metadata

metadata:
  - version: 0.2
    description: "GEdit-Bench with detailed metrics by language (en/cn) and subset (fullset/intersection)"

Design Patterns

Hierarchical Metric Organization

The configuration demonstrates a three-level metric hierarchy: 1. Global level: Overall metrics across all samples 2. Language level: Separate tracking for English and Chinese 3. Subset level: Distinction between fullset (all samples) and intersection (overlapping subset)

This allows fine-grained performance analysis across linguistic and categorical boundaries.

Multiple Score Dimensions

Each category tracks three score types:

Semantics: Semantic preservation and accuracy
Quality: Generation quality metrics
Overall: Combined score

Custom Aggregation Functions

Each metric uses a dedicated aggregation function, allowing specialized processing for different language and subset combinations while maintaining consistent metric naming conventions.

Related Components

Utility functions: `lmms_eval/tasks/gedit_bench/utils.py`
Dataset: `stepfun-ai/GEdit-Bench` on Hugging Face Hub
Principle: YAML_Task_Configuration

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment