Implementation:EvolvingLMMs Lab Lmms eval Lmms Is Simple

Knowledge Sources	lmms-eval
Domains	Model_Architecture, Evaluation
Last Updated	2026-02-14 00:00 GMT

Overview

Concrete tool for selecting between simple and chat model protocols provided by the lmms-eval framework.

Description

The is_simple class-level boolean attribute on the lmms base class controls which task protocol a model uses during evaluation. When is_simple = True (the default), the evaluator builds tasks in "simple" mode and passes raw visuals via doc_to_visual. When is_simple = False, tasks are built in "chat" mode and deliver structured ChatMessages via doc_to_messages.

The _validate_model_class static method in ModelRegistryV2 enforces that the resolved model type (simple or chat) matches the is_simple flag on the loaded class. A mismatch raises a TypeError with a clear message indicating which attribute needs to be changed or which class path needs to be registered.

After model instantiation, the evaluator reads lm.is_simple to determine task_type, which is passed to get_task_dict() to ensure tasks provide data in the expected format.

Usage

Use this attribute when creating a new model class that extends lmms. Set it at class definition time as a class variable. The default is True, so only chat-style models need to explicitly set is_simple = False.

Code Reference

Source Location

• Repository: lmms-eval
• File: lmms_eval/api/model.py
• Lines: L26-27

Validation Source

• File: lmms_eval/models/registry_v2.py
• Lines: L120-137

Signature

class lmms(abc.ABC):
    is_simple: bool = True

Validation Logic

@staticmethod
def _validate_model_class(cls: type, resolved: ResolvedModel) -> None:
    from lmms_eval.api.model import lmms

    if not (isinstance(cls, type) and issubclass(cls, lmms)):
        raise TypeError(
            f"Model class '{resolved.class_path}' is not a subclass of lmms",
        )

    cls_is_simple = getattr(cls, "is_simple", True)
    if resolved.model_type == "chat" and cls_is_simple:
        raise TypeError(
            f"Model '{resolved.model_id}' resolved as chat but "
            f"{cls.__name__}.is_simple is True. "
            f"Set is_simple = False on the class.",
        )
    if resolved.model_type == "simple" and not cls_is_simple:
        raise TypeError(
            f"Model '{resolved.model_id}' resolved as simple but "
            f"{cls.__name__}.is_simple is False. "
            f"Set is_simple = True or register a chat_class_path.",
        )

Import

from lmms_eval.api.model import lmms

I/O Contract

Inputs

Name	Type	Required	Description
is_simple	bool	No (defaults to True)	Class-level attribute. True for simple models that receive raw visuals via doc_to_visual. False for chat models that receive structured ChatMessages via doc_to_messages.

Outputs

Name	Type	Description
task_type	str	Either "simple" or "chat", derived from lm.is_simple after instantiation. Passed to get_task_dict() to build tasks with the correct input protocol.
Instance args format (simple)	tuple	(contexts, gen_kwargs, doc_to_visual, doc_id, task, split)
Instance args format (chat)	tuple	(ctx, doc_to_messages, gen_kwargs, doc_id, task, split)

Usage Examples

Simple Model (Default)

from lmms_eval.api.model import lmms

class MySimpleModel(lmms):
    # is_simple defaults to True, so no need to set it explicitly
    def __init__(self, pretrained: str, device: str = "cuda", batch_size: int = 1, **kwargs):
        super().__init__()
        self.model = load_model(pretrained)

    def generate_until(self, requests):
        results = []
        for req in requests:
            contexts, gen_kwargs, doc_to_visual, doc_id, task, split = req.args
            visuals = doc_to_visual(req.doc)
            output = self.model.generate(contexts, visuals, **gen_kwargs)
            results.append(output)
        return results

Chat Model

from lmms_eval.api.model import lmms

class MyChatModel(lmms):
    is_simple = False  # Explicitly set for chat protocol

    def __init__(self, pretrained: str, device: str = "cuda", batch_size: int = 1, **kwargs):
        super().__init__()
        self.model = load_model(pretrained)

    def generate_until(self, requests):
        results = []
        for req in requests:
            ctx, doc_to_messages, gen_kwargs, doc_id, task, split = req.args
            chat_messages = doc_to_messages(req.doc)
            hf_messages = chat_messages.to_hf_messages()
            output = self.model.chat(hf_messages, **gen_kwargs)
            results.append(output)
        return results

Related Pages

Implements Principle

• Principle:EvolvingLMMs_Lab_Lmms_eval_Model_Type_Selection