Implementation:EvolvingLMMs Lab Lmms eval Lmms Base Class
| Knowledge Sources | |
|---|---|
| Domains | Model_Architecture, Evaluation |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
Concrete tool for wrapping multimodal models with generation and loglikelihood methods provided by the lmms-eval framework.
Description
The lmms class in lmms_eval/api/model.py is the abstract base class that all model integrations must extend. It defines three abstract methods (generate_until, loglikelihood, generate_until_multi_round) and provides infrastructure for instance creation, caching, distributed execution, and resource cleanup.
Subclasses must implement at least the abstract methods for the request types their target tasks use. The evaluator dispatches requests to the model by calling getattr(lm, reqtype)(cloned_reqs), where reqtype matches the task's output type.
The class also provides create_from_arg_string, a class method that parses the --model_args CLI string into keyword arguments and merges them with additional config (batch_size, device, max_batch_size) before calling the subclass constructor.
Usage
Use this class as the base for any custom model integration. Implement the abstract methods to connect the framework's request-response pattern to your model's inference logic.
Code Reference
Source Location
- Repository: lmms-eval
- File:
lmms_eval/api/model.py - Lines: L26-331
Signature
class lmms(abc.ABC):
is_simple: bool = True
def __init__(self) -> None: ...
@abc.abstractmethod
def loglikelihood(self, requests: List[Instance]) -> List[Tuple[float, bool]]: ...
@abc.abstractmethod
def generate_until(self, requests) -> List[str]: ...
@abc.abstractmethod
def generate_until_multi_round(self, requests) -> List[str]: ...
@classmethod
def create_from_arg_string(
cls: Type[T],
arg_string: str,
additional_config: Optional[dict] = None,
) -> T: ...
@property
def rank(self) -> int: ...
@property
def world_size(self) -> int: ...
def clean(self) -> None: ...
Import
from lmms_eval.api.model import lmms
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| requests | List[Instance] |
Yes | List of Instance objects. Each Instance carries request_type, arguments (a tuple whose format depends on simple/chat protocol), doc_id, task_name, and repeats.
|
| arg_string | str |
Yes (for create_from_arg_string) |
Comma-separated key=value pairs, e.g. "pretrained=Qwen/Qwen2.5-VL-7B,max_pixels=12845056".
|
| additional_config | Optional[dict] |
No | Additional constructor arguments such as batch_size, device, max_batch_size. None values are filtered out.
|
Outputs
| Name | Type | Description |
|---|---|---|
| generate_until result | List[str] |
One generated text string per input request. |
| loglikelihood result | List[Tuple[float, bool]] |
One (log_probability, is_greedy) tuple per input request. log_probability is the log-likelihood of the continuation. is_greedy indicates whether the continuation matches greedy decoding.
|
| generate_until_multi_round result | List[str] |
One generated text string per multi-round request. |
| create_from_arg_string result | T |
An instantiated model object of the subclass type. |
Usage Examples
Basic Custom Model
import torch
from typing import List, Tuple
from lmms_eval.api.model import lmms
from lmms_eval.api.instance import Instance
class MyVisionModel(lmms):
is_simple = True
def __init__(
self,
pretrained: str,
device: str = "cuda",
batch_size: int = 1,
**kwargs,
):
super().__init__()
self.device = device
self.batch_size = int(batch_size)
self.model = AutoModelForCausalLM.from_pretrained(pretrained).to(device)
self.processor = AutoProcessor.from_pretrained(pretrained)
def generate_until(self, requests: List[Instance]) -> List[str]:
results = []
for req in requests:
contexts, gen_kwargs, doc_to_visual, doc_id, task, split = req.args
visuals = doc_to_visual(req.doc)
inputs = self.processor(text=contexts, images=visuals, return_tensors="pt").to(self.device)
output_ids = self.model.generate(**inputs, **gen_kwargs)
text = self.processor.decode(output_ids[0], skip_special_tokens=True)
results.append(text)
return results
def loglikelihood(self, requests: List[Instance]) -> List[Tuple[float, bool]]:
# Return placeholder if not supported
return [(0.0, False)] * len(requests)
def generate_until_multi_round(self, requests: List[Instance]) -> List[str]:
# Delegate to single-round for simplicity
return self.generate_until(requests)
Instantiation via CLI
# The evaluator calls this internally:
model_cls = models.get_model("my_vision_model")
lm = model_cls.create_from_arg_string(
"pretrained=my-org/my-model,max_pixels=12845056",
{"batch_size": 8, "max_batch_size": None, "device": "cuda:0"},
)