Workflow:Open compass VLMEvalKit Adding Custom VLM
| Knowledge Sources | |
|---|---|
| Domains | VLM_Evaluation, Model_Integration, Development |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
Process for integrating a new Vision-Language Model into VLMEvalKit so it can be evaluated across all supported benchmarks.
Description
This workflow guides developers through adding support for a new VLM to VLMEvalKit. The core requirement is implementing a single generate_inner() method in a model adapter class. The adapter handles converting VLMEvalKit's standardized multi-modal message format into the model-specific input format, running inference, and returning the text prediction. Optional extensions include custom prompt building per dataset and multi-turn chat support. Once integrated, the model can be evaluated on all 70+ image and video benchmarks without additional per-benchmark work.
Usage
Execute this workflow when you want to evaluate a VLM that is not yet supported by VLMEvalKit. You should have access to the model weights or API, understand the model's input format requirements, and be familiar with Python class inheritance.
Execution Steps
Step 1: Create Model Adapter Class
Create a new Python file in vlmeval/vlm/ (for local models) or vlmeval/api/ (for API models). Define a class that inherits from BaseModel (for local models) or BaseAPI (for API models). These base classes handle prompt building, image processing, message format conversion, and retry logic.
Key considerations:
- Local model adapters go in vlmeval/vlm/ and extend BaseModel
- API model adapters go in vlmeval/api/ and extend BaseAPI
- Set INTERLEAVE = False if the model does not support interleaved images and text
- The class __init__ should handle model loading (weights, tokenizer, processor)
Step 2: Implement generate_inner Method
Implement the mandatory generate_inner(msgs, dataset=None) method. This method receives a list of multi-modal message dictionaries and returns the model's text prediction as a string. Each message dictionary has a type (either "image" or "text") and a value (file path/URL for images, string for text).
What happens:
- The method receives standardized multi-modal messages
- Convert messages to the model's expected input format
- Run model inference (forward pass for local, HTTP request for API)
- Return the generated text as a single string
- The optional dataset argument can trigger dataset-specific behavior
Step 3: Implement Custom Prompt Building (Optional)
Optionally implement use_custom_prompt(dataset) and build_prompt(line, dataset) to customize how prompts are constructed for specific benchmarks. This is useful when a model has dataset-specific prompt templates that improve evaluation accuracy.
Key considerations:
- use_custom_prompt returns a boolean indicating whether to use custom prompts for a given dataset
- build_prompt receives a data sample and returns a multi-modal message list
- If not implemented, VLMEvalKit uses the default dataset-level prompt construction
- Model-level build_prompt takes precedence over dataset-level build_prompt
Step 4: Implement Multi-Turn Chat Support (Optional)
Optionally implement chat_inner(message, dataset) for models that support multi-turn conversation. This enables evaluation on multi-turn benchmarks like MMDU.
Key considerations:
- The message parameter is a list of conversation turns with role and content keys
- Roles alternate between "user" and "assistant"
- The last turn is always a user turn
- Return the model's response as a single string
Step 5: Register Model in Config
Register the new model in vlmeval/config.py by adding an entry to the appropriate model dictionary. The entry maps a model name string to a functools.partial() constructor call. Also add the import in vlmeval/vlm/__init__.py (or vlmeval/api/__init__.py for API models).
Key considerations:
- The model name string becomes the identifier used with --model in run.py
- Use functools.partial() to pre-configure constructor arguments (model path, parameters)
- Group the entry with similar models in the config dictionaries
- Add multiple entries for different model sizes/variants
Step 6: Validate and Test
Test the integration by running vlmutil check {model_name} which loads the model and performs a quick inference test. Then run a small benchmark evaluation to verify end-to-end correctness.
Key considerations:
- vlmutil check validates that the model loads and generates output
- Test on a small benchmark first (e.g., a dataset with few samples)
- Verify that the output format is compatible with the evaluation pipeline
- Run pre-commit checks before submitting a contribution