Implementation:EvolvingLMMs Lab Lmms eval Baseline Registry

Knowledge Sources	EvolvingLMMs_Lab_Lmms_eval
Domains	Evaluation, Model Comparison
Last Updated	2026-02-14 00:00 GMT

Overview

The Baseline Registry is a centralized dictionary mapping model names to their baseline evaluation results across different tasks. It provides a structured way to reference baseline results by model and task name, supporting both local paths and HuggingFace Hub URLs.

Description

The registry is a two-dimensional dictionary keyed first by model identifier and then by task name. Each model entry may include an optional _meta dictionary for metadata (model name, default HuggingFace repository) and one or more task entries containing either a HuggingFace URL or a local filesystem path to a JSONL baseline file. Keys starting with an underscore are filtered out when listing available tasks, providing a clean convention for separating metadata from task entries.

Usage

Use the Baseline Registry when comparing new model results against established baselines. Pass a model preset name via the --baseline CLI argument and the framework automatically resolves the corresponding baseline file for the current task.

# Auto-match current task
python -m lmms_eval --model new_model --tasks videomme --baseline qwen25vl

# Explicitly specify task
python -m lmms_eval --model new_model --tasks mmbench --baseline qwen25vl:videomme

Code Reference

Source Location

• Repository: EvolvingLMMs-Lab/lmms-eval
• File: lmms_eval/baselines/registry.py
• Lines: 1--44

Key Components

BASELINE_REGISTRY

BASELINE_REGISTRY: Dict[str, Dict[str, Any]] = {
    "qwen25vl": {
        "_meta": {
            "model": "Qwen2.5-VL-7B-Instruct",
            "hf_repo": "mwxely/lmms-eval-test",
        },
        "videomme": {
            "hf_url": "hf://mwxely/lmms-eval-test/20251111_202127_samples_videomme_w_subtitle.jsonl",
            "description": "VideoMME w/ subtitle",
        },
    },
}

Purpose: Central mapping of model presets to their baseline results across benchmarks.

Structure:

• Top-level keys -- Model identifiers (e.g., "qwen25vl")
• _meta -- Optional metadata dictionary containing human-readable model name and default HuggingFace repository
• Task keys -- Task-specific baseline entries containing an hf_url or path and an optional description

_meta Dictionary

"_meta": {
    "model": "Full Model Name",
    "hf_repo": "default/hf/repo",
}

Purpose: Store model-level metadata that is not a task entry.

Fields:

• model -- Human-readable model name
• hf_repo -- Default HuggingFace repository for this model's baselines

Task Entry

"task_name": {
    "hf_url": "hf://user/repo/file.jsonl",
    "description": "Description of baseline",
}

Purpose: Point to a specific baseline JSONL file for a given task.

Required (one of):

• hf_url -- HuggingFace Hub URL in hf://user/repo/path/file.jsonl format
• path -- Local filesystem path to baseline JSONL file

Optional:

• description -- Human-readable description of the baseline

I/O Contract

Input	Type	Description
model_name	str	Top-level key identifying the model preset
task_name	str	Second-level key identifying the task

Output	Type	Description
task_entry	Dict[str, Any]	Dictionary with hf_url or path and optional description

Usage Patterns

Accessing the Registry

from lmms_eval.baselines.registry import BASELINE_REGISTRY

# Check if model exists
if "qwen25vl" in BASELINE_REGISTRY:
    model_entry = BASELINE_REGISTRY["qwen25vl"]

# Get metadata
meta = model_entry.get("_meta", {})
model_name = meta.get("model")

# List available tasks (exclude _meta)
available_tasks = [k for k in model_entry.keys() if not k.startswith("_")]

Adding a New Baseline

BASELINE_REGISTRY["gpt4v"] = {
    "_meta": {
        "model": "GPT-4V",
        "hf_repo": "openai/gpt4v-baselines",
    },
    "videomme": {
        "hf_url": "hf://openai/gpt4v-baselines/videomme_results.jsonl",
        "description": "GPT-4V on VideoMME benchmark",
    },
    "mmbench": {
        "path": "/shared/baselines/gpt4v_mmbench.jsonl",
        "description": "GPT-4V on MMBench",
    },
}

Integration with Loader

from lmms_eval.baselines.registry import BASELINE_REGISTRY

def load_baseline(baseline_arg: str, task_name: str):
    if baseline_arg in BASELINE_REGISTRY:
        return _load_from_registry(baseline_arg, task_name, baseline_arg)

def _load_from_registry(model_name: str, task_name: str, baseline_arg: str):
    model_entry = BASELINE_REGISTRY[model_name]
    if task_name not in model_entry:
        available_tasks = [k for k in model_entry.keys() if not k.startswith("_")]
        raise ValueError(
            f"No baseline for model '{model_name}' on task '{task_name}'. "
            f"Available tasks: {available_tasks}"
        )
    task_entry = model_entry[task_name]
    if "hf_url" in task_entry:
        return _load_baseline_from_hf(task_entry["hf_url"], task_name)
    elif "path" in task_entry:
        return _load_baseline_from_local(task_entry["path"], task_name)

Design Decisions

• Two-dimensional structure -- Natural organization by model and task allows easy addition of new models or new tasks for existing models.
• Underscore prefix for metadata -- Simple convention to distinguish metadata from task entries without complex filtering logic.
• Flexible URL/path storage -- Supports both cloud (HuggingFace) and local storage without prescribing one approach.
• Dictionary type -- Simple, readable, and easy to extend without complex class hierarchies.
• Default HF repo in metadata -- Reduces duplication when multiple tasks use the same repository.

Related Pages

• Principle: EvolvingLMMs_Lab_Lmms_eval_Baseline_Comparison
• See Also: EvolvingLMMs_Lab_Lmms_eval_Baseline_Loader