Implementation:Mit han lab Llm awq QuantInternVisionEncoder

Knowledge Sources	Mit_han_lab_Llm_awq
Domains	Quantization, Vision
Last Updated	2026-02-15 00:00 GMT

Overview

Concrete tool for running InternViT vision encoder layers with W8A8 (8-bit weight, 8-bit activation) quantization using fused CUDA kernels.

Description

QuantInternVisionEncoder wraps the standard InternVisionEncoder layers with quantized counterparts. QuantInternAttention replaces attention projections with W8A8OF16LinearDynamicInputScale layers and uses fused QKV computation. QuantInternMLP quantizes the feed-forward layers. QuantInternRMSNorm implements RMS normalization with per-token quantization support using awq_inference_engine. QuantInternVisionEncoderLayer composes these into a full quantized encoder block with residual connections and layer scaling.

Usage

Import QuantInternVisionEncoder when deploying InternVL3 models with W8A8 vision encoder quantization for reduced memory and faster inference.

Code Reference

Source Location

Repository: Mit_han_lab_Llm_awq
File: tinychat/modules/fused_internencoder.py
Lines: 1-237

Signature

class QuantInternVisionEncoder(nn.Module):
    def __init__(self, module: InternVisionEncoder, bsz=64, seqlen=1024):
        """Wrap InternVisionEncoder layers with quantized versions."""
    def forward(self, inputs_embeds, attention_mask=None, output_attentions=None,
                output_hidden_states=None, return_dict=None) -> BaseModelOutput: ...

class QuantInternRMSNorm(nn.Module):
    def __init__(self, module: nn.Module, use_per_token_quant=True): ...
    def forward(self, hidden_states) -> Tuple[torch.Tensor, torch.Tensor]: ...

class QuantInternAttention(nn.Module):
    def __init__(self, module: InternAttention, config: InternVisionConfig, init_only=False): ...
    def forward(self, hidden_states: torch.Tensor, scale_in: torch.Tensor) -> torch.Tensor: ...

class QuantInternMLP(nn.Module):
    def __init__(self, module: InternMLP, config: InternVisionConfig): ...
    def forward(self, hidden_states: torch.Tensor, scale_in: torch.Tensor) -> torch.Tensor: ...

class QuantInternVisionEncoderLayer(nn.Module):
    def __init__(self, module: InternVisionEncoderLayer, config: InternVisionConfig): ...
    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor: ...

Import

from tinychat.modules.fused_internencoder import QuantInternVisionEncoder

I/O Contract

Inputs

Name	Type	Required	Description
module	InternVisionEncoder	Yes	Pre-trained encoder to quantize
bsz	int	No	Maximum batch size for buffer allocation (default: 64)
seqlen	int	No	Maximum sequence length for buffer allocation (default: 1024)
inputs_embeds	torch.Tensor	Yes	Input embeddings from vision patch encoding

Outputs

Name	Type	Description
forward returns	BaseModelOutput	Encoder output with last_hidden_state

Usage Examples

Quantize InternViT Encoder

from tinychat.modules.fused_internencoder import QuantInternVisionEncoder

# Wrap pre-trained encoder with quantized version
quant_encoder = QuantInternVisionEncoder(
    model.vision_model.encoder, bsz=1, seqlen=1025
)
# Replace in model
model.vision_model.encoder = quant_encoder

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment