Principle:EvolvingLMMs Lab Lmms eval Model Wrapper Implementation

Knowledge Sources	lmms-eval
Domains	Model_Architecture, Evaluation
Last Updated	2026-02-14 00:00 GMT

Overview

Implementing the abstract model interface for multimodal evaluation requires defining generation and loglikelihood methods that process batched requests.

Description

The lmms abstract base class defines the contract that every model integration must fulfill. It specifies three abstract methods that correspond to the core evaluation capabilities: free-form text generation, log-likelihood computation, and multi-round dialogue generation.

generate_until: The primary method for open-ended generation tasks. It receives a list of Instance objects, each containing a prompt context, generation parameters, and visual/message inputs. The method must return a list of generated strings, one per request. This method powers tasks with output type "generate_until", which is the most common evaluation mode for multimodal models.

loglikelihood: Used for multiple-choice and scoring tasks where the model must compute the probability of a given continuation given a context. It returns a list of (float, bool) tuples representing the log probability and whether the continuation would be the greedy-decoded output. Tasks with output type "loglikelihood" or "multiple_choice" dispatch through this method.

generate_until_multi_round: An extension of generate_until for tasks that require multi-turn dialogue. The method receives requests that may include conversation history, and the model must maintain state across turns within a single request.

Beyond these abstract methods, the base class provides concrete infrastructure for:

• Instance creation via create_from_arg_string, which parses comma-separated key-value model arguments (e.g., pretrained=Qwen/Qwen2.5-VL-7B,max_pixels=12845056) and passes them to the subclass constructor.
• Distributed execution with rank and world-size properties that default to single-process but can be overridden for multi-GPU setups.
• Response caching through JSONL-based cache files that store model outputs keyed by task name and document ID, enabling evaluation resumption.
• Resource cleanup via the clean() method which releases GPU memory by deleting all nn.Module attributes and calling torch.cuda.empty_cache().

Usage

Subclass lmms when creating any new model integration for the lmms-eval framework. At minimum, implement generate_until. Implement loglikelihood if the model supports token-level probability computation. Implement generate_until_multi_round if the model will be used with multi-turn evaluation tasks.

Theoretical Basis

The model wrapper pattern follows the Strategy pattern from object-oriented design: the framework defines an interface (the abstract base class) and delegates the model-specific implementation to concrete subclasses. This separation allows the evaluation pipeline to be model-agnostic while supporting diverse model architectures.

The request-response flow follows this sequence:

1. Evaluator builds task instances
2. Instances are grouped by request type (generate_until, loglikelihood, etc.)
3. Grouped instances are passed to the corresponding model method
4. Model processes instances (potentially in batches) and returns results
5. Results are attached back to their originating instances
6. Task post-processing and metric computation follow

The Instance dataclass carries all information needed for a single evaluation request:

Instance:
  request_type: "generate_until" | "loglikelihood" | "generate_until_multi_round"
  arguments: tuple  (protocol-specific: simple or chat format)
  idx: int          (ordering index)
  metadata: dict    (task_name, doc_id, repeats)
  resps: list       (filled by model)
  filtered_resps: dict  (filled by post-processing filters)

The constructor pattern uses create_from_arg_string to bridge CLI arguments to Python constructor calls:

CLI: --model_args pretrained=X,max_pixels=12845056
  -> simple_parse_args_string("pretrained=X,max_pixels=12845056")
  -> {"pretrained": "X", "max_pixels": "12845056"}
  -> cls(pretrained="X", max_pixels="12845056", batch_size=..., device=...)

Related Pages

Implemented By

• Implementation:EvolvingLMMs_Lab_Lmms_eval_Lmms_Base_Class