Implementation:Vllm project Vllm VLM Registry Lookup

Knowledge Sources	vLLM vLLM Docs
Domains	Vision Language Models, Model Selection, Multimodal AI
Last Updated	2026-02-08 13:00 GMT

Overview

Concrete tool for selecting a supported vision-language model from the vLLM model registry and its companion example map, provided by vLLM.

Description

vLLM maintains two complementary registries for VLM selection:

1. _MULTIMODAL_EXAMPLE_MODELS in tests/models/registry.py: A dictionary mapping architecture class names (e.g., LlavaForConditionalGeneration) to _HfExamplesInfo dataclass instances containing the default HuggingFace model ID, trust settings, transformers version constraints, and other metadata.
2. model_example_map in examples/offline_inference/vision_language.py: A dictionary mapping short model type keys (e.g., "llava", "qwen2_vl", "internvl_chat") to runner functions that return complete ModelRequestData including engine args and prompt templates.

Each runner function encapsulates the model-specific configuration: the HuggingFace model name, engine arguments (max_model_len, max_num_seqs, trust_remote_code, enforce_eager, tensor_parallel_size), prompt templates with the correct vision token placeholders, and stop token IDs.

Usage

Use the VLM registry lookup when:

• Selecting a model for offline VLM inference with vLLM.
• Verifying that a specific VLM architecture is supported by the current vLLM version.
• Looking up the default configuration and prompt template for a given VLM.

Code Reference

Source Location

• Repository: vllm
• File: tests/models/registry.py (architecture registry), examples/offline_inference/vision_language.py (example map)

Signature

# Architecture registry entry
@dataclass(frozen=True)
class _HfExamplesInfo:
    default: str                    # Default HuggingFace model ID
    extras: Mapping[str, str]       # Additional model variants
    trust_remote_code: bool = False
    enforce_eager: bool = False
    max_model_len: int | None = None
    hf_overrides: dict[str, Any] = field(default_factory=dict)
    ...

# Example map: short name -> runner function
model_example_map: dict[str, Callable] = {
    "llava": run_llava,
    "qwen2_vl": run_qwen2_vl,
    "internvl_chat": run_internvl,
    ...  # 60+ entries
}

# Runner function return type
class ModelRequestData(NamedTuple):
    engine_args: EngineArgs
    prompts: list[str]
    stop_token_ids: list[int] | None = None
    lora_requests: list[LoRARequest] | None = None
    sampling_params: list[SamplingParams] | None = None

Import

from tests.models.registry import HF_EXAMPLE_MODELS, _MULTIMODAL_EXAMPLE_MODELS

I/O Contract

Inputs

Name	Type	Required	Description
model_type	str	Yes	Short key identifying the VLM (e.g., "llava", "qwen2_5_vl", "phi3_v")
questions	list[str]	Yes	List of text questions/prompts to pair with visual input
modality	str	Yes	Input modality: "image", "video", or "vision_chunk"

Outputs

Name	Type	Description
engine_args	EngineArgs	Fully configured engine arguments for the selected model
prompts	list[str]	Formatted prompts with model-specific vision token placeholders
stop_token_ids	None	Model-specific stop token IDs for generation termination

Usage Examples

Selecting LLaVA-1.5 for Image Understanding

from examples.offline_inference.vision_language import model_example_map

questions = ["What is the content of this image?"]
req_data = model_example_map["llava"](questions, "image")

# req_data.engine_args.model == "llava-hf/llava-1.5-7b-hf"
# req_data.prompts == ["USER: <image>\nWhat is the content of this image?\nASSISTANT:"]

Selecting Qwen2.5-VL for Video Analysis

questions = ["Why is this video funny?"]
req_data = model_example_map["qwen2_5_vl"](questions, "video")

# req_data.engine_args.model == "Qwen/Qwen2.5-VL-3B-Instruct"
# Prompt includes <|vision_start|><|video_pad|><|vision_end|> tokens

Selecting InternVL for Multi-purpose VQA

questions = ["Describe the content of this image in detail."]
req_data = model_example_map["internvl_chat"](questions, "image")

# req_data.engine_args.model == "OpenGVLab/InternVL3-2B"
# req_data.engine_args.trust_remote_code == True
# Uses tokenizer.apply_chat_template for prompt formatting

Checking Architecture Support in Registry

from tests.models.registry import _MULTIMODAL_EXAMPLE_MODELS

# Check if an architecture is supported
assert "LlavaForConditionalGeneration" in _MULTIMODAL_EXAMPLE_MODELS
assert "Qwen2VLForConditionalGeneration" in _MULTIMODAL_EXAMPLE_MODELS

# Get default model for an architecture
info = _MULTIMODAL_EXAMPLE_MODELS["Phi3VForCausalLM"]
print(info.default)  # "microsoft/Phi-3-vision-128k-instruct"
print(info.trust_remote_code)  # True

Related Pages

Implements Principle

• Principle:Vllm_project_Vllm_VLM_Model_Selection