Principle:Sgl project Sglang Model Quantization And Loading
| Knowledge Sources | |
|---|---|
| Domains | Quantization, Model_Loading, Model_Optimization |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
A factory-pattern model loading mechanism that selects the appropriate loader based on quantization settings and orchestrates the quantize-export pipeline.
Description
Model quantization and loading involves selecting the correct model loader (standard or ModelOpt), loading the base model, applying quantization algorithms, and optionally exporting the quantized model to disk. SGLang uses a factory function get_model_loader that inspects the configuration to determine whether standard loading or ModelOpt-based quantization is needed. The ModelOpt loader handles the complete pipeline: load base model → apply quantization (via modelopt.torch.quantization) → save checkpoint → export safetensors.
Usage
Use the model loader factory when performing post-training quantization or loading pre-quantized models. The factory automatically selects the correct loader based on the quantization setting in ModelConfig.
Theoretical Basis
The quantization pipeline follows a calibration-free PTQ approach:
- Load base model in full precision (FP16/BF16)
- Apply quantization config (e.g., FP8_DEFAULT_CFG)
- Quantize weights using ModelOpt's mtq.quantize or weight conversion
- Export quantized weights as safetensors with updated config.json
Factory pattern rationale:
- Different quantization methods require different loading strategies
- Pre-quantized models need direct loading (no re-quantization)
- Standard models need the default safetensors/pytorch loader