Implementation:EvolvingLMMs Lab Lmms eval Evaluation Shell Script
| Knowledge Sources | |
|---|---|
| Domains | Testing, Evaluation |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
Concrete tool for creating reproducible model evaluation scripts provided by the lmms-eval framework.
Description
The evaluation shell script pattern combines the HuggingFace Accelerate launcher with lmms-eval CLI arguments to create a single, self-contained command that reproduces an evaluation run. The canonical example in the repository is examples/models/qwen25vl.sh, which demonstrates multi-GPU evaluation of the Qwen2.5-VL model on the MME benchmark.
The script sets environment variables for cache configuration, then invokes accelerate launch with process count and port arguments, followed by -m lmms_eval and the full set of model, task, and batch parameters.
Usage
Use this pattern when documenting evaluation commands for a newly integrated model. Place the script in examples/models/<model_name>.sh and include comments explaining any model-specific arguments.
Code Reference
Source Location
- Repository: lmms-eval
- File:
examples/models/qwen25vl.sh - Lines: L1-18
Signature
accelerate launch --num_processes=N --main_process_port=PORT -m lmms_eval \
--model <model_name> \
--model_args=<key1=value1,key2=value2,...> \
--tasks <task1,task2,...> \
--batch_size N
Import
# No import needed -- this is a shell script pattern.
# Ensure lmms-eval is installed:
pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| --num_processes | int |
Yes | Number of GPU processes to spawn via Accelerate. Set to 1 for single-GPU evaluation. |
| --main_process_port | int |
No | Communication port for distributed processes. Default chosen by Accelerate. Useful to avoid port conflicts when running multiple evaluations. |
| --model | str |
Yes | Registered model name (e.g., qwen2_5_vl, llava_hf, internvl2).
|
| --model_args | str |
Yes | Comma-separated key=value pairs passed to the model constructor: pretrained=<hub_id>, max_pixels=<N>, attn_implementation=<type>, etc.
|
| --tasks | str |
Yes | Comma-separated task names or task groups (e.g., mme, mmmu,mmlu_flan_n_shot_generative).
|
| --batch_size | int |
Yes | Number of samples per batch. Use 1 for models that do not support batched inference.
|
Outputs
| Name | Type | Description |
|---|---|---|
| Console output | text | Evaluation results table printed to stdout with per-task metrics. |
| Results file | JSON | When --output_path is specified, aggregated results and per-sample logs are saved to the given directory.
|
Usage Examples
Multi-GPU Evaluation (Reference Script)
#!/bin/bash
# Evaluate Qwen2.5-VL-7B on MME with 8 GPUs
export HF_HOME="~/.cache/huggingface"
accelerate launch --num_processes=8 --main_process_port=12346 -m lmms_eval \
--model qwen2_5_vl \
--model_args=pretrained=Qwen/Qwen2.5-VL-7B-Instruct,max_pixels=12845056,attn_implementation=flash_attention_2,interleave_visuals=False \
--tasks mme \
--batch_size 1
Single-GPU Evaluation
#!/bin/bash
# Single-GPU evaluation with output logging
python -m lmms_eval \
--model qwen2_5_vl \
--model_args pretrained=Qwen/Qwen2.5-VL-3B-Instruct,max_pixels=12845056,attn_implementation=sdpa \
--tasks mmmu,mme,mmlu_flan_n_shot_generative \
--batch_size 128 \
--limit 8 \
--device cuda:0
Custom Model Evaluation Script
#!/bin/bash
# Evaluate a custom model integration
export HF_HOME="/data/hf_cache"
accelerate launch --num_processes=4 --main_process_port=29500 -m lmms_eval \
--model my_custom_model \
--model_args=pretrained=my-org/my-model-v1,max_pixels=8000000 \
--tasks mme,mmmu \
--batch_size 4 \
--log_samples \
--output_path ./results/my_custom_model/
Force Simple Protocol
#!/bin/bash
# Force simple protocol for a model that has both simple and chat implementations
python -m lmms_eval \
--model qwen2_5_vl \
--model_args pretrained=Qwen/Qwen2.5-VL-7B-Instruct \
--tasks mme \
--batch_size 1 \
--force_simple