Workflow:Turboderp org Exllamav2 Multimodal Vision Inference
| Knowledge Sources | |
|---|---|
| Domains | LLMs, Inference, Vision_Language_Models, Multimodal |
| Last Updated | 2026-02-15 00:00 GMT |
Overview
End-to-end process for performing vision-language inference using ExLlamaV2 with supported multimodal models (Pixtral, Qwen2-VL, Gemma3, Mistral Small 3.1) to generate text descriptions or answers based on image inputs.
Description
This workflow demonstrates how to use ExLlamaV2's vision-language model (VLM) support to process images and generate text responses. It covers loading the vision tower (image encoder), processing images into visual embeddings, constructing multimodal prompts with image placeholders, and generating text that references the visual content. The vision tower handles image preprocessing, patch extraction, and encoding through architectures like SigLIP, Pixtral, or Qwen2-VL vision encoders, then projects the visual embeddings into the language model's embedding space via a multimodal projector.
Usage
Execute this workflow when you have a supported vision-language model (Pixtral-12B, Qwen2-VL-7B, Gemma3, or Mistral Small 3.1) and need to generate text based on one or more input images. Use cases include image captioning, visual question answering, image comparison, object identification, and document understanding. Images can be loaded from local files or URLs.
Execution Steps
Step 1: Model_Configuration
Initialize the model configuration from the vision-language model directory. Set an appropriate maximum sequence length (VLMs like Pixtral default to 1M tokens, which should be reduced for practical use). The configuration parser auto-detects the VLM architecture and sets up the required parameters for both the vision and language components.
Key considerations:
- Pixtral defaults to 1M token context; reduce to practical size (e.g., 8192)
- The model directory must contain both vision and language model weights
- Architecture detection handles Pixtral, Qwen2-VL, SigLIP/Gemma3 variants
- Max sequence length should account for image token expansion
Step 2: Vision_Tower_Loading
Load the vision tower (image encoder) and multimodal projector from the model directory. The vision tower contains the image processing pipeline specific to the model architecture: Pixtral uses its own vision transformer, Qwen2-VL uses a specialized vision encoder, and Gemma3/Mistral use SigLIP. The multimodal projector maps vision encoder outputs to the language model's embedding dimension.
Key considerations:
- The vision tower must be loaded before the language model
- Each VLM architecture has its own vision encoder implementation
- The projector bridges the dimension gap between vision and language embeddings
- Progress tracking is available during loading
Step 3: Language_Model_Loading
Load the main language model with a KV-cache using auto-split, and initialize the tokenizer. This follows the standard model loading procedure. The language model processes both text tokens and injected visual embeddings through its transformer layers.
Key considerations:
- Standard auto-split loading with lazy cache allocation
- Cache size should account for the expanded token count from images
- The tokenizer must support the model's special tokens for image placeholders
Step 4: Image_Processing
Load images from files or URLs and convert them to visual embeddings using the vision tower. Each image is processed through the vision encoder and projector to produce an embedding object containing the visual token representations and a text alias (placeholder string). The embeddings are assigned unique temporary token IDs that persist for the lifetime of the process.
Key considerations:
- Images can be loaded from local files (PIL) or URLs (via requests)
- Each image embedding gets a unique text alias for prompt insertion
- Embeddings can be reused across multiple generations for efficiency
- The generator applies prompt caching and deduplication to image tokens
Step 5: Multimodal_Prompt_Construction
Construct a prompt that combines image placeholders with text instructions using the model-specific chat template. Insert the image embedding aliases at the appropriate positions in the prompt according to the VLM's expected format. Different models use different prompt structures (Pixtral uses INST tags, Qwen2-VL uses im_start/im_end, Gemma3 uses start_of_turn/end_of_turn).
Key considerations:
- Prompt format varies by model family and must match exactly
- Multiple images can be included in a single prompt
- Image aliases are replaced with temporary token IDs during encoding
- Special tokens must be encoded as tokens (encode_special_tokens = True)
Step 6: Vision_Language_Generation
Generate text by passing the encoded prompt and image embeddings to the Dynamic Generator. The model processes both text and visual tokens through its forward pass, attending to both modalities to produce a coherent text response. Generation can be performed in batch mode (collect full output) or streaming mode (token-by-token via DynamicJob).
Key considerations:
- Both batch and streaming generation modes are supported
- Embeddings must be passed to both the tokenizer encode call and the job/generate call
- Stop conditions should match the model's expected end-of-turn tokens
- Greedy sampling is often preferred for factual image descriptions