Workflow:Sgl project Sglang ModelOpt Quantization And Export
| Knowledge Sources | |
|---|---|
| Domains | LLM_Inference, Quantization, Model_Optimization |
| Last Updated | 2026-02-09 00:00 GMT |
Overview
End-to-end process for quantizing a language model using NVIDIA ModelOpt (FP8 or FP4) and exporting it for optimized deployment with SGLang.
Description
This workflow covers quantizing a pre-trained model to reduced precision (FP8 or FP4) using NVIDIA's ModelOpt toolkit, exporting the quantized weights in a format compatible with SGLang, and deploying the quantized model for inference. Quantization reduces memory footprint and increases inference throughput by leveraging hardware-accelerated low-precision computation on NVIDIA GPUs (Hopper, Blackwell architectures). The workflow automates the full pipeline from loading a full-precision model to serving the quantized result.
Usage
Execute this workflow when you want to reduce the memory requirements of a model for deployment, increase inference throughput, or deploy on hardware with limited GPU memory. FP8 quantization is recommended for Hopper (H100) and newer GPUs; FP4 for Blackwell (B200/GB200) GPUs. This workflow requires NVIDIA ModelOpt installed and CUDA-capable GPU hardware.
Execution Steps
Step 1: Initialize Distributed Environment
Set up the minimal distributed computing environment required for model loading. Even for single-GPU quantization, the distributed environment must be initialized because SGLang's model loader uses the distributed infrastructure for weight sharding and communication.
Key considerations:
- Set RANK, WORLD_SIZE, MASTER_ADDR, MASTER_PORT environment variables
- Use NCCL backend for GPU communication
- For multi-GPU quantization, configure tensor and pipeline parallelism sizes
Step 2: Configure Model and Quantization Settings
Create the model configuration specifying the source model path and the desired quantization method. Configure the load settings with optional checkpoint save path and export directory. SGLang supports "modelopt_fp8" and "modelopt_fp4" quantization methods.
Key considerations:
- model_path: HuggingFace hub ID or local model directory
- quantization: "modelopt_fp8" for 8-bit or "modelopt_fp4" for 4-bit floating point
- Export directory will contain the complete quantized model ready for serving
- Optional checkpoint save path preserves the intermediate ModelOpt state
Step 3: Load and Quantize the Model
Use SGLang's model loader to load the original model weights, apply ModelOpt quantization calibration, and transform the weights to the target precision. The loader automatically handles weight mapping, quantization scale computation, and format conversion.
Key considerations:
- The model loader detects ModelOpt quantization and invokes the appropriate pipeline
- Calibration uses representative data to compute optimal quantization scales
- Memory usage during quantization may be higher than the final model size
- Export happens automatically after successful quantization
Step 4: Validate the Export
Verify that the exported model directory contains all required files: config.json, tokenizer configuration, and quantized model weight files in safetensors format. The validation step confirms that the export is complete and the quantization configuration is properly recorded.
Key considerations:
- Check for config.json, tokenizer_config.json, and model weight files
- Verify that quantization_config is present in the exported config
- Ensure all model shards are present for large models
Step 5: Deploy the Quantized Model
Launch a new SGLang Engine or server pointing to the exported model directory with the "modelopt" quantization flag. SGLang auto-detects whether the exported model uses FP8 or FP4 from the configuration and loads the appropriate kernels.
Key considerations:
- Use quantization="modelopt" for auto-detection of FP4/FP8
- The quantized model requires less GPU memory than the original
- All standard serving features (batching, caching, streaming) work with quantized models
- Performance gains depend on GPU architecture and model size