Implementation:Mit han lab Llm awq Load awq llama fast

Overview

Concrete tool for efficiently loading AWQ-quantized weights into TinyChat models provided by the llm-awq library.

Source

File: tinychat/utils/load_quant.py, Lines 142-171

Signature

def load_awq_llama_fast(model, checkpoint, w_bit, group_size, device):
    ...

Import

from tinychat.utils.load_quant import load_awq_llama_fast

I/O

Inputs

• model (nn.Module) - TinyChat model instance
• checkpoint (str) - path to .pt or .safetensors file
• w_bit (int) - weight bit width, 4
• group_size (int) - quantization group size, 128
• device (str) - target device, e.g. "cuda:0"

Output

• nn.Module with WQLinear layers loaded on the target device

Details

• Supports both .pt (via torch.load) and .safetensors (via safetensors.torch.load_file) formats
• Also supports mem_efficient_load mode for per-tensor shard loading

Related Pages

• Principle:Mit_han_lab_Llm_awq_Quantized_Weight_Loading
• Environment:Mit_han_lab_Llm_awq_CUDA_Build_Environment

Knowledge Sources

• Repo|llm-awq|https://github.com/mit-han-lab/llm-awq

Domains

• Inference
• Deployment