Principle:EvolvingLMMs Lab Lmms eval Registry Registration

Knowledge Sources	lmms-eval
Domains	Model_Management, Configuration
Last Updated	2026-02-14 00:00 GMT

Overview

Registering model implementations in a typed registry with lazy imports and alias support enables the framework to discover and load models by name.

Description

The lmms-eval framework uses a dual-registry system to map model names to their implementations. Understanding this system is essential for making a custom model discoverable by the CLI and evaluation pipeline.

Legacy Registry (MODEL_REGISTRY): The original registry uses the @register_model("name") decorator, which eagerly imports the model class and stores it in a global dictionary. When the decorator is applied, it validates that the class is a subclass of lmms and registers it under one or more alias names. This approach works but requires importing all model modules at startup.

V2 Registry (MODEL_REGISTRY_V2): The modern registry uses ModelManifest dataclasses and ModelRegistryV2 for lazy loading. Instead of importing classes eagerly, manifests store dotted class paths (e.g., "lmms_eval.models.simple.qwen2_5_vl.Qwen2_5_VL") that are only imported when the model is actually requested. This dramatically reduces startup time for the 80+ built-in models.

A ModelManifest contains:

• model_id: The canonical model name (e.g., "qwen2_5_vl").
• simple_class_path: Dotted path to the simple-protocol implementation class, or None.
• chat_class_path: Dotted path to the chat-protocol implementation class, or None.
• aliases: Additional names that resolve to this model.

At least one of simple_class_path or chat_class_path must be provided.

Resolution semantics: When a model is requested, ModelRegistryV2.resolve() prefers the chat implementation unless force_simple=True is set. If only one implementation exists, that one is used regardless of the force_simple flag.

Registration sources:

1. Built-in models: Defined in AVAILABLE_SIMPLE_MODELS and AVAILABLE_CHAT_TEMPLATE_MODELS dicts in lmms_eval/models/__init__.py, converted to manifests at import time.
2. Entry points: External packages can register models via the lmms_eval.models entry point group.
3. Legacy plugins: The deprecated LMMS_EVAL_PLUGINS environment variable for backward compatibility.

Usage

To register a new model:

• For built-in models: Add entries to AVAILABLE_SIMPLE_MODELS and/or AVAILABLE_CHAT_TEMPLATE_MODELS in lmms_eval/models/__init__.py. Place the implementation at the expected module path (lmms_eval/models/simple/<name>.py or lmms_eval/models/chat/<name>.py).
• For external plugins: Define a ModelManifest and expose it via the lmms_eval.models entry point in your package's pyproject.toml.
• For quick prototyping: Use the @register_model("name") decorator, which immediately populates the legacy registry.

Theoretical Basis

The registry follows the Service Locator pattern, where a central registry maps string identifiers to service implementations. The V2 registry adds lazy loading, making it closer to a Dependency Injection container with deferred instantiation.

The resolution algorithm:

resolve(model_name, force_simple=False):
    1. Look up model_id from alias table
    2. Retrieve ModelManifest for model_id
    3. If force_simple AND simple_class_path exists:
         -> return ResolvedModel(type="simple", path=simple_class_path)
    4. Elif chat_class_path exists:
         -> return ResolvedModel(type="chat", path=chat_class_path)
    5. Else (only simple exists):
         -> return ResolvedModel(type="simple", path=simple_class_path)

The class loading sequence:

get_model_class(model_name):
    1. resolved = resolve(model_name)
    2. module = importlib.import_module(resolved.module_path)
    3. cls = getattr(module, resolved.class_name)
    4. _validate_model_class(cls, resolved)  # check is_simple consistency
    5. return cls

The manifest merging supports incremental registration -- a model can be registered first with only a simple path, then later extended with a chat path (or vice versa) without conflicts:

register("my_model", simple_class_path="pkg.simple.MyModel")
register("my_model", chat_class_path="pkg.chat.MyModel")
# Result: manifest has both paths populated

Related Pages

Implemented By

• Implementation:EvolvingLMMs_Lab_Lmms_eval_Register_Model