Workflow:Huggingface Alignment handbook QLoRA Single GPU Finetuning
| Knowledge Sources | |
|---|---|
| Domains | LLMs, Fine_Tuning, Parameter_Efficient_Training |
| Last Updated | 2026-02-07 00:00 GMT |
Overview
End-to-end process for parameter-efficient fine-tuning of language models using QLoRA (4-bit quantized LoRA) on a single consumer GPU, enabling SFT and DPO alignment of 7B+ models with limited hardware.
Description
This workflow enables the full SFT → DPO alignment pipeline on consumer hardware by combining 4-bit NormalFloat quantization with Low-Rank Adaptation (LoRA). Instead of updating all model weights, QLoRA freezes the base model in 4-bit precision and trains small adapter matrices injected into attention and feedforward layers. This reduces GPU memory requirements from ~80GB to under 24GB, allowing training of 7B-parameter models on a single RTX 4090 or equivalent. The workflow produces lightweight LoRA adapter weights that can be uploaded to and downloaded from the Hugging Face Hub efficiently.
Usage
Execute this workflow when you need to fine-tune a 7B+ parameter model but have access to only a single GPU with 24GB or less VRAM. This is ideal for rapid prototyping, custom domain adaptation, or situations where full fine-tuning hardware is unavailable. The resulting adapter weights are small and fast to share, making this approach practical for iterative experimentation.
Execution Steps
Step 1: Environment Setup
Install the alignment-handbook package, PyTorch with CUDA support, Flash Attention 2, and the bitsandbytes library for 4-bit quantization. Configure the accelerate environment for single-GPU training using the DDP config with num_processes set to 1.
Key considerations:
- bitsandbytes is required for 4-bit quantization support
- The DDP accelerate config with a single process is the correct choice for single-GPU training
- Flash Attention 2 provides significant speedup even on single GPU
Step 2: QLoRA Configuration
Create or select a YAML recipe config that specifies the LoRA parameters alongside the standard training configuration. Key LoRA parameters include the rank (r), alpha scaling factor, dropout rate, and target modules. The config must enable 4-bit loading and PEFT training.
Key considerations:
- Typical LoRA rank (r) is 16, with alpha equal to r for unit scaling
- Target modules for transformer models include q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj
- A dropout of 0.05 on LoRA layers provides regularization
- The learning rate for QLoRA is typically 10x higher than full fine-tuning (e.g., 2e-4 vs 2e-5)
Step 3: Model Loading with Quantization
Load the base model in 4-bit NormalFloat precision using bitsandbytes integration. The model loading utility applies the quantization config automatically when load_in_4bit is enabled, and sets up the appropriate device map for quantized inference.
Key considerations:
- 4-bit NF4 quantization is applied to all linear layers
- The quantization device map ensures layers are placed correctly on the GPU
- Use cache is disabled when gradient checkpointing is active
Step 4: SFT Training with LoRA
Run supervised fine-tuning using the SFTTrainer with PEFT config enabled. The trainer automatically injects LoRA adapter layers into the specified target modules, freezes the base model weights, and trains only the adapter parameters. The chat template is applied to format the training data.
Key considerations:
- Only LoRA adapter parameters (~0.1-1% of total) are updated during training
- Gradient accumulation compensates for smaller batch sizes on single GPU
- Per-device batch size is typically 4 (vs 16 for full fine-tuning)
- TensorBoard is the default logger for single-GPU training
Step 5: DPO Training with LoRA
Align the SFT adapter model with preferences using DPOTrainer with PEFT config. The reference model can share the base weights, reducing memory further. Training continues to update only the adapter weights while the quantized base model remains frozen.
Key considerations:
- The SFT adapter output serves as the starting point for DPO
- Reference model memory can be reduced by sharing the base model weights
- Beta and learning rate may need adjustment compared to full fine-tuning
Step 6: Adapter Saving and Publishing
Save the trained LoRA adapter weights and push to the Hugging Face Hub. The adapter files are typically only a few hundred MB compared to the full model's tens of GB, enabling fast upload and sharing.
Key considerations:
- Only adapter weights are saved (not the full base model)
- Adapter weights can be loaded on top of any compatible base model
- The Hub model ID follows the pattern {username}/{model_name}-{task}