Heuristic:Huggingface Trl QLoRA BF16 Adapter Casting
| Knowledge Sources | |
|---|---|
| Domains | Optimization, Quantization, Parameter_Efficient_Finetuning |
| Last Updated | 2026-02-06 17:00 GMT |
Overview
When using QLoRA (4-bit or 8-bit quantized models with LoRA), cast trainable adapter parameters to bfloat16 for numerical stability and performance as recommended by the original QLoRA paper.
Description
The QLoRA paper recommends training adapter weights in bfloat16 precision even when the base model is quantized to 4-bit or 8-bit. This ensures numerical stability during the forward and backward passes. TRL automatically detects quantized models (via is_loaded_in_4bit or is_loaded_in_8bit attributes) and casts all trainable parameters to torch.bfloat16. This is a workaround because PEFT's autocast_adapter_dtype=False option is not yet supported for quantized models.
Usage
Apply this heuristic whenever using QLoRA (4-bit/8-bit quantized base model + LoRA adapters). TRL applies this automatically in GRPOTrainer, DPOTrainer, and other trainers. If building a custom training loop with quantized models, manually cast trainable params to bf16.
The Insight (Rule of Thumb)
- Action: Cast all trainable (LoRA adapter) parameters to
torch.bfloat16when the base model is loaded in 4-bit or 8-bit. - Value:
param.data = param.data.to(torch.bfloat16)for allparam.requires_grad == True. - Trade-off: Slight reduction in precision compared to fp32, but matches the QLoRA paper's recommendations and provides better training stability than fp16.
Reasoning
Quantized models have their frozen weights in 4-bit or 8-bit, but the computations during forward/backward passes are done in higher precision. The LoRA adapter weights (the trainable parameters) need to be in a precision that balances memory savings with numerical stability. BFloat16 provides the same dynamic range as fp32 (unlike fp16) while using half the memory, making it the ideal choice for adapter training on quantized models.
Code evidence from `trl/trainer/grpo_trainer.py:338-346`:
# When using QLoRA, the PEFT adapter weights are converted to bf16 to follow the
# recommendations from the original paper (see https://huggingface.co/papers/2305.14314,
# paragraph 3). Normally, this can be done by passing `autocast_adapter_dtype=False` to
# `get_peft_model`, but this option is not yet supported for quantized models.
# See: https://github.com/huggingface/peft/issues/2889
# Non-quantized models do not have the `is_loaded_in_{8,4}bit` attributes
if getattr(model, "is_loaded_in_4bit", False) or getattr(model, "is_loaded_in_8bit", False):
for param in model.parameters():
if param.requires_grad:
param.data = param.data.to(torch.bfloat16)