Principle:Open compass VLMEvalKit Model Validation
| Source | Domain | Last Updated |
|---|---|---|
| Repo | Vision, Testing | 2026-02-14 00:00 GMT |
Overview
A validation pattern that tests whether a newly registered VLM correctly loads, generates predictions, and reports its capabilities.
Description
After registering a model, VLMEvalKit provides vlmutil check (backed by the CHECK() function) to validate the integration. CHECK instantiates the model from the registry, runs test generations with different input formats (plain text, dict format, single image, multiple images), and reports the model's declared capabilities (INTERLEAVE support, allowed input types, VIDEO_LLM flag). This ensures the adapter correctly implements the BaseModel/BaseAPI contract before running expensive benchmark evaluations.
Usage
Run vlmutil check model_name after adding a new model adapter. Verifies both basic generation and capability reporting.
Theoretical Basis
Smoke testing / integration testing — verify that the adapter correctly implements the interface contract by running representative test cases.