Workflow:Huggingface Diffusers Model Quantization
| Knowledge Sources | |
|---|---|
| Domains | Diffusion_Models, Quantization, Optimization |
| Last Updated | 2026-02-13 21:00 GMT |
Overview
End-to-end process for reducing diffusion model memory footprint through weight quantization using multiple backend frameworks.
Description
This workflow covers loading and running diffusion models with reduced numerical precision to decrease VRAM requirements and potentially improve inference speed. Diffusers supports five quantization backends: BitsAndBytes (4-bit NF4/FP4 and 8-bit LLM.int8), GGUF (pre-quantized format), TorchAO (PyTorch-native int4/int8/float8), Quanto (float8/int8/int4/int2), and NVIDIA ModelOpt. The workflow includes selecting the appropriate backend, configuring quantization parameters, loading models with on-the-fly quantization, and running inference with quantized components. Pipeline-level quantization allows applying different quantization configs to different sub-models (e.g., 4-bit transformer with full-precision VAE).
Usage
Execute this workflow when you need to run a diffusion model but have limited GPU memory, or when you want to reduce the memory footprint to enable larger batch sizes or higher resolutions. This is especially relevant for large transformer-based models (Flux, SD3, HunyuanVideo) that require significant VRAM in full precision. Quantization trades a small amount of output quality for substantial memory savings, enabling models that would otherwise require enterprise GPUs to run on consumer hardware.
Execution Steps
Step 1: Backend Selection
Choose the quantization backend based on your hardware, model architecture, and quality requirements. Each backend offers different tradeoffs between compression ratio, inference speed, quality preservation, and hardware compatibility.
Key considerations:
- BitsAndBytes: Most mature, supports 4-bit (NF4 best quality) and 8-bit, CUDA-only
- TorchAO: Native PyTorch, supports int4/int8/float8, works with torch.compile
- Quanto: Flexible bit-widths down to int2, cross-platform
- GGUF: Pre-quantized checkpoint format, no on-the-fly quantization needed
- ModelOpt: NVIDIA-specific optimizations for TensorRT deployment
- 4-bit quantization reduces model size by approximately 4x with minimal quality loss
Step 2: Quantization Configuration
Create a quantization configuration object specifying the backend-specific parameters. This configuration defines the precision level, computation dtype, calibration settings, and which layers to quantize or skip.
Key considerations:
- BitsAndBytes 4-bit: choose between NF4 (better quality) and FP4 quant types
- Double quantization (bnb_4bit_use_double_quant) further reduces memory with minimal overhead
- Compute dtype should match your training/inference dtype (typically float16 or bfloat16)
- TorchAO supports multiple quantization layouts including int4_weight_only and float8_dynamic_activation
- Certain layers (normalization, embeddings) may need to be excluded from quantization
Step 3: Model Loading with Quantization
Load the diffusion model components with quantization applied during the loading process. The quantization config is passed to the model loader which applies weight compression on-the-fly as weights are loaded from disk. This avoids ever materializing the full-precision model in memory.
Key considerations:
- Pass quantization_config to from_pretrained() for individual model loading
- For pipeline-level quantization, use PipelineQuantizationConfig to specify per-component configs
- The transformer/UNet is typically the primary quantization target
- The VAE should generally remain in full precision to preserve image quality
- Text encoders can be quantized with less quality impact than the denoising model
Step 4: Pipeline Assembly
Assemble the quantized model components into a complete inference pipeline. Combine the quantized denoising model with full-precision VAE, text encoders, and scheduler. Configure memory optimizations that are compatible with quantized models.
Key considerations:
- Not all memory optimizations are compatible with quantized models
- CPU offloading works with quantized models but may be slower due to dtype conversions
- Attention slicing is generally compatible with all quantization backends
- Some quantization backends support torch.compile for additional speedup
- Verify the pipeline produces acceptable quality before deploying
Step 5: Quantized Inference
Run inference using the quantized pipeline. The quantized model dequantizes weights on-the-fly during computation, performing the actual matrix multiplications in the compute dtype before re-quantizing. The inference API is identical to the non-quantized pipeline.
Key considerations:
- First inference call may be slower due to kernel compilation (especially with TorchAO)
- Quality should be compared against full-precision outputs to verify acceptable degradation
- Memory savings can be verified using torch.cuda.max_memory_allocated()
- Batch size can often be increased thanks to the reduced memory footprint
- LoRA adapters can be applied on top of quantized models
Step 6: Saving and Sharing
Optionally serialize and save the quantized model for reuse without repeating the quantization process. Some backends support saving the quantized state directly, while others require re-quantization on each load.
Key considerations:
- BitsAndBytes quantized models can be serialized and shared via the Hub
- GGUF models are inherently pre-quantized and stored in that format
- TorchAO and Quanto models may need re-quantization on load depending on the configuration
- Include quantization metadata in the model card for reproducibility