Implementation:EvolvingLMMs Lab Lmms eval Get Model

Knowledge Sources	lmms-eval
Domains	Evaluation, Model_Management
Last Updated	2026-02-14 00:00 GMT

Overview

Concrete tool for resolving model names to instantiated model objects via a typed registry with lazy imports, provided by the lmms-eval framework.

Description

The get_model() function is the public entry point for model resolution. It delegates to ModelRegistryV2.get_model_class(), which performs a three-step process: (1) resolves the model name to a canonical ModelManifest via alias lookup, (2) selects the appropriate class path based on model type preference (chat over simple), and (3) lazily imports the class via importlib.import_module() and validates it as a subclass of lmms.

The returned class is then instantiated by calling create_from_arg_string(), which parses the CLI --model_args string (e.g., pretrained=Qwen/Qwen2.5-VL-3B-Instruct,max_pixels=12845056) into keyword arguments passed to the model's constructor.

The registry is initialized at module import time by scanning two dictionaries -- AVAILABLE_SIMPLE_MODELS and AVAILABLE_CHAT_TEMPLATE_MODELS -- and building manifests for each unique model ID. Plugin models can be registered via Python entry points in the lmms_eval.models group.

Usage

Use get_model() when you need to:

Resolve a model name to its class for instantiation in the evaluation pipeline.
Force simple mode for a model that has both simple and chat implementations.
Inspect the available model manifest for debugging registration issues.

Code Reference

Source Location

Repository: lmms-eval
File: lmms_eval/models/__init__.py (L221-227), lmms_eval/models/registry_v2.py (L82-118), lmms_eval/api/model.py (L291-306)

Signature

# Top-level entry point
def get_model(model_name: str, force_simple: bool = False) -> type:
    """Resolve model name to class via ModelRegistryV2."""
    return MODEL_REGISTRY_V2.get_model_class(
        model_name, force_simple=force_simple
    )

# Registry resolution
class ModelRegistryV2:
    def get_model_class(
        self, model_name: str, force_simple: bool = False
    ) -> type: ...

# Instance creation from argument string
class lmms(abc.ABC):
    @classmethod
    def create_from_arg_string(
        cls: Type[T],
        arg_string: str,
        additional_config: Optional[dict] = None,
    ) -> T: ...

Import

from lmms_eval.models import get_model

I/O Contract

Inputs

Name	Type	Required	Description
model_name	str	Yes	Model identifier string (e.g., `"qwen2_5_vl"`, `"llava_onevision"`)
force_simple	bool	No	If True, forces the simple (non-chat) implementation even if a chat variant exists (default: False)
arg_string	str	No	Comma-separated key=value pairs for model constructor (e.g., `"pretrained=Qwen/Qwen2.5-VL-3B-Instruct,max_pixels=12845056"`)
additional_config	Optional[dict]	No	Extra configuration dictionary merged into constructor kwargs

Outputs

Name	Type	Description
model_class	type	A Python class that is a subclass of `lmms`, with `is_simple` flag set appropriately
model_instance	T (subclass of lmms)	Instantiated model object (returned by `create_from_arg_string`)

Usage Examples

Basic Example

from lmms_eval.models import get_model

# Resolve model name to class
model_cls = get_model("qwen2_5_vl")

# Instantiate with arguments
lm = model_cls.create_from_arg_string(
    "pretrained=Qwen/Qwen2.5-VL-3B-Instruct,"
    "max_pixels=12845056,"
    "attn_implementation=sdpa"
)

print(type(lm))       # <class 'lmms_eval.models.chat.qwen2_5_vl.Qwen2_5_VL'>
print(lm.is_simple)   # False (chat mode preferred)

Force Simple Mode

from lmms_eval.models import get_model

# Force simple (non-chat) mode
model_cls = get_model("qwen2_5_vl", force_simple=True)
lm = model_cls.create_from_arg_string(
    "pretrained=Qwen/Qwen2.5-VL-3B-Instruct"
)
print(lm.is_simple)   # True

List Available Models

from lmms_eval.models import list_available_models

# List all canonical model IDs
models = list_available_models()
print(models[:5])
# ['aero', 'aria', 'audio_flamingo_3', 'auroracap', 'bagel']

# Include aliases
all_names = list_available_models(include_aliases=True)

Related Pages

Implements Principle

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment