Principle:EvolvingLMMs Lab Lmms eval Model Configuration

Knowledge Sources	lmms-eval
Domains	Evaluation, Model_Management
Last Updated	2026-02-14 00:00 GMT

Overview

Model configuration is the process of resolving a model name string to a fully instantiated model object through a typed registry, lazy imports, and argument parsing.

Description

Multi-modal evaluation frameworks must support a wide variety of model architectures -- from locally loaded Hugging Face checkpoints to cloud-hosted API models -- behind a uniform interface. Model configuration solves this by providing a registry-based dispatch mechanism that maps short model identifiers (like qwen2_5_vl) to concrete Python classes implementing the lmms abstract base class.

The lmms-eval framework uses a two-tier model registry (V2):

1. ModelManifest -- A frozen dataclass declaring a model's canonical ID, an optional simple-mode class path, and an optional chat-mode class path. Manifests are the declarative registration unit.
2. ModelRegistryV2 -- The runtime registry that stores manifests, resolves aliases, handles model type preference (chat over simple), and performs lazy imports of model classes.

Each model class may expose two variants:

• Simple mode (is_simple = True) -- The model receives raw text prompts and visual inputs. The framework builds the full prompt string.
• Chat mode (is_simple = False) -- The model receives structured message lists. The framework delegates prompt formatting to the model's chat template.

The registry prefers the chat implementation when available, unless force_simple=True is specified. Model classes are imported lazily -- only when actually requested -- to avoid loading heavy dependencies (torch, transformers) for models that are not used.

Usage

Use model configuration whenever:

• You are launching an evaluation and need to specify which model to use via --model and --model_args.
• You are adding a new model to the framework and need to register it in the model registry.
• You need to programmatically discover which models are available via list_available_models().
• You are developing a plugin model and want to register it via Python entry points.

Theoretical Basis

The model resolution follows a three-step dispatch chain:

Step 1 -- Registry Lookup:

Given a model name m, find the canonical ModelManifest:

• Check direct ID match: m in manifests
• Check alias match: alias_to_model_id[m]
• If neither matches, raise ValueError with available names.

Step 2 -- Type Resolution:

Given the manifest and a force_simple flag:

• If force_simple and simple_class_path exists, select simple mode.
• Else if chat_class_path exists, select chat mode.
• Else select simple mode (guaranteed to exist by manifest validation).

Step 3 -- Lazy Import and Validation:

The class_path (e.g., lmms_eval.models.chat.qwen2_5_vl.Qwen2_5_VL) is split into module_path and class_name, imported via importlib.import_module(), and validated:

• Must be a subclass of lmms.
• Its is_simple flag must be consistent with the resolved model type.

The class is then instantiated via create_from_arg_string(), which parses the comma-separated model_args string into keyword arguments.

Related Pages

Implemented By

• Implementation:EvolvingLMMs_Lab_Lmms_eval_Get_Model