Implementation:Mit han lab Llm awq InternVL3
| Knowledge Sources | |
|---|---|
| Domains | Vision, NLP, Multimodal |
| Last Updated | 2026-02-15 00:00 GMT |
Overview
Concrete tool for performing AWQ-quantized multimodal inference combining an InternViT vision encoder with a LLaMA or Qwen2 language model provided by the tinychat framework.
Description
The InternVL3 class composes InternVisionModel for visual feature extraction and a quantized LLM (LlamaForCausalLM or Qwen2ForCausalLM) for text generation. Vision features are extracted via extract_features(), which runs the ViT, removes the CLS token, applies pixel shuffle downsampling, and projects through an MLP bridge (mlp1). The _embed() method fuses visual and text embeddings by replacing IMG_CONTEXT token positions in the text embedding with vision features. It supports streaming generation (stream_gen), benchmarking, and both image and video inputs. Weight initialization is skipped for faster loading.
Usage
Import this class when deploying InternVL3 models for quantized multimodal inference. Use extract_features() to process images, _embed() to fuse vision and text, and stream_gen() or forward() for generation.
Code Reference
Source Location
- Repository: Mit_han_lab_Llm_awq
- File: tinychat/models/internvl3.py
- Lines: 1-383
Signature
class InternVL3(PreTrainedModel):
config_class = InternVLChatConfig
main_input_name = 'pixel_values'
base_model_prefix = 'language_model'
_supports_flash_attn_2 = True
supports_gradient_checkpointing = True
_no_split_modules = ['InternVisionModel', 'LlamaDecoderLayer', 'Qwen2DecoderLayer']
def __init__(
self,
config: InternVLChatConfig,
vision_model=None,
language_model=None,
use_flash_attn=True,
):
"""
Args:
config: InternVLChatConfig with vision and language model settings.
vision_model: Optional pre-built vision model.
language_model: Optional pre-built language model.
use_flash_attn: Whether to enable Flash Attention.
"""
def extract_features(self, pixel_values) -> torch.Tensor:
"""Extract and project vision features through ViT + MLP bridge."""
def _embed(self, input_ids, media, media_config, labels, attention_mask):
"""Fuse visual and text embeddings by replacing image tokens."""
def stream_gen(self, input_ids, media, media_cfg, start_pos,
chunk_prefilling, quant_llm, attention_mask=None):
"""Streaming token generation for quantized or FP16 inference."""
def benchmark(self, prompt, quant_llm) -> str:
"""Benchmark model with timing measurements."""
def forward(
self,
pixel_values: torch.FloatTensor,
input_ids: torch.LongTensor = None,
attention_mask: Optional[torch.Tensor] = None,
position_ids: Optional[torch.LongTensor] = None,
image_flags: Optional[torch.LongTensor] = None,
past_key_values: Optional[List[torch.FloatTensor]] = None,
labels: Optional[torch.LongTensor] = None,
use_cache: Optional[bool] = None,
output_attentions: Optional[bool] = None,
output_hidden_states: Optional[bool] = None,
return_dict: Optional[bool] = None,
) -> Union[Tuple, CausalLMOutputWithPast]:
"""Full forward pass combining vision and language models."""
Import
from tinychat.models.internvl3 import InternVL3
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| pixel_values | torch.FloatTensor | Yes | Image tensors of shape (batch, num_patches, channels, height, width) |
| input_ids | torch.LongTensor | Yes | Tokenized input with IMG_CONTEXT placeholders |
| attention_mask | torch.Tensor | No | Attention mask for padded sequences |
| image_flags | torch.LongTensor | No | Flags indicating which images are real vs padding |
| past_key_values | List[torch.FloatTensor] | No | KV cache for autoregressive generation |
Outputs
| Name | Type | Description |
|---|---|---|
| logits | torch.FloatTensor | Language model output logits |
| past_key_values | Tuple | Updated KV cache for next generation step |
| extract_features returns | torch.Tensor | Vision features of shape (batch, num_tokens, llm_hidden_size) |
| stream_gen returns | Tuple[Any, int] | Generated token and updated position |
Usage Examples
Multimodal Inference
from tinychat.models.internvl3 import InternVL3
from tinychat.models.internvl.configuration_internvl import InternVLChatConfig
from tinychat.models.internvl.media import load_image
# Load model
config = InternVLChatConfig.from_pretrained("path/to/internvl3")
model = InternVL3(config).cuda().half()
# Load and process image
pixel_values = load_image("photo.jpg", input_size=448)
pixel_values = pixel_values.cuda().half()
# Extract vision features
vision_features = model.extract_features(pixel_values)
# Run benchmark
response = model.benchmark("Describe this image.", quant_llm=None)