Principle:Mit han lab Llm awq Quantized Weight Loading
Overview
Process of constructing a model architecture with quantized linear layers and loading pre-quantized INT4 weights from checkpoint files.
Description
Loading a quantized model requires two steps:
- Replace all nn.Linear layers in the model with WQLinear shells (empty quantized modules with the correct shapes)
- Load the checkpoint state dict containing packed INT4 weights, scales, and zeros into these modules
This avoids the memory cost of loading FP16 weights first. Two approaches exist:
- load_awq_model uses accelerate's dispatch for multi-GPU deployment
- load_awq_llama_fast directly replaces layers and loads the state dict for single-GPU use
Usage
When deploying a pre-quantized AWQ model for inference (TinyChat, serving).
Related Pages
Knowledge Sources
- Repo|llm-awq|https://github.com/mit-han-lab/llm-awq
Domains
- Inference
- Deployment
Page Connections
Double-click a node to navigate. Hold to expand connections.
Principle
Implementation
Heuristic
Environment