Principle:Turboderp org Exllamav2 Image Embedding Extraction
| Knowledge Sources | |
|---|---|
| Domains | Vision_Language_Models, Image_Processing, Deep_Learning |
| Last Updated | 2026-02-15 00:00 GMT |
Overview
Image embedding extraction converts raw images into sequences of embedding vectors in the language model's hidden space, enabling the model to attend to visual information during text generation.
Description
To process images within a language model, the visual content must be transformed into a representation the language model can work with. This transformation is a multi-stage pipeline:
Stage 1: Image Preprocessing The raw image is prepared according to the specific vision encoder's requirements. This includes resizing to the expected resolution, normalizing pixel values, and extracting patches. Different architectures handle this differently:
- Pixtral supports variable-resolution images with dynamic patch grids
- Qwen2-VL uses dynamic resolution with specific aspect ratio handling
- SigLIP uses fixed-resolution resizing with standard normalization
Stage 2: Vision Encoder Forward Pass The preprocessed image tensor is passed through the vision tower's transformer layers. Each patch embedding attends to all other patches through self-attention, producing contextualized visual feature vectors. The output is a sequence of N feature vectors of dimension d_vision.
Stage 3: Multimodal Projection The vision features are projected through the multimodal projector to match the language model's hidden dimension. This maps the N vectors from d_vision to d_language dimensions.
Stage 4: Embedding Container Creation The projected embeddings are wrapped in an ExLlamaV2MMEmbedding container that also stores:
- A text_alias (placeholder string like "<image>") used in the prompt
- Allocated token IDs that will be replaced with the actual embeddings during the forward pass
- The embeddings tensor itself, of shape (num_tokens, hidden_size)
Usage
Use image embedding extraction whenever you need to include image content in a prompt for a vision-language model. Each image processed through this pipeline produces an embedding container that is then referenced in the text prompt through its placeholder alias.
Theoretical Basis
The embedding extraction process follows this pipeline:
Input: PIL Image (H x W x 3)
1. Preprocess:
- Resize/pad to target resolution
- Normalize: pixel = (pixel - mean) / std
- Extract patches: (H/P) x (W/P) patches of size P x P
- Flatten to sequence: N = (H/P) * (W/P) patch vectors
2. Vision Encoder:
- Input: patch_embeddings ∈ R^{N × d_vision}
- For each transformer layer:
z = LayerNorm(x)
z = MultiHeadAttention(z) + x (residual)
z = LayerNorm(z)
z = MLP(z) + z (residual)
- Output: features ∈ R^{N × d_vision}
3. Multimodal Projector:
- Linear/MLP mapping: d_vision → d_language
- Output: embeddings ∈ R^{N × d_language}
4. Container:
- Allocate N token IDs from reserved range
- Map text_alias → token ID range
- Store embeddings for injection during forward pass
The number of output tokens N depends on the image resolution and patch size. For example, a 336x336 image with 14x14 patches produces 24x24 = 576 tokens, each representing a spatial region of the image.