Principle:Mit han lab Llm awq Checkpoint Preparation
Overview
Process of converting and splitting quantized model checkpoints into formats optimized for efficient loading on deployment targets.
Description
AWQ quantized checkpoints may need format conversion or splitting depending on the deployment scenario. Two primary operations are involved:
(1) Checkpoint Splitting breaks a single large .pt file into per-tensor shard files, enabling memory-efficient loading where each tensor is loaded individually rather than loading the entire checkpoint into memory.
(2) Weight Repacking converts v1-format checkpoints (sequential layout) to v2-format (interleaved qweight, transposed scales, precomputed scaled_zeros) required by newer CUDA kernels for optimal inference performance.
Usage
When preparing quantized checkpoints for TinyChat deployment, especially on memory-constrained devices.
Related Pages
Metadata
- Knowledge Sources: Repo|llm-awq|https://github.com/mit-han-lab/llm-awq
- Domains: Deployment, Model_Compression