Principle:Sgl project Sglang Model Quantization Configuration
| Knowledge Sources | |
|---|---|
| Domains | Quantization, Model_Optimization, Configuration |
| Last Updated | 2026-02-10 00:00 GMT |
Overview
A configuration pattern that specifies model architecture, quantization method, and export paths for post-training quantization workflows.
Description
Model quantization configuration defines the settings needed to load a pre-trained model, apply quantization (FP8 or FP4), and export the quantized result. In SGLang, this involves two dataclasses: ModelConfig (which specifies the model path, architecture, and quantization method) and LoadConfig (which specifies loading format, export paths, and checkpoint locations). Together, they provide all the information needed for the model loading and quantization pipeline.
Usage
Configure model quantization settings when performing post-training quantization with NVIDIA ModelOpt. This is the setup step before the actual quantization and export process.
Theoretical Basis
Post-training quantization (PTQ) reduces model weights from higher precision (FP16/BF16) to lower precision (FP8/FP4):
- FP8 (8-bit floating point): Reduces memory by ~2x, with minimal accuracy loss for most models
- FP4 (4-bit floating point): Reduces memory by ~4x, may require calibration for accuracy
Configuration requirements:
- Model path — Where to load the original weights from
- Quantization method — Which quantization scheme to apply (e.g., "modelopt_fp8", "modelopt_fp4")
- Export path — Where to save the quantized model
- Checkpoint path — Optional intermediate checkpoint for resumption