Principle:Intel Ipex llm LoRA Adapter Injection QLoRA

Knowledge Sources	LoRA: Low-Rank Adaptation QLoRA IPEX-LLM
Domains	NLP, Parameter_Efficient_Finetuning
Last Updated	2026-02-09 00:00 GMT

Overview

Technique for injecting trainable low-rank adapter matrices into a 4-bit quantized model for parameter-efficient fine-tuning.

Description

LoRA (Low-Rank Adaptation) injects pairs of low-rank decomposition matrices (A and B) into selected transformer layers. For QLoRA specifically, the base model is frozen at 4-bit precision while the LoRA adapters are trained in bfloat16. The process involves: (1) preparing the quantized model for k-bit training by freezing base parameters and enabling gradient computation on adapters, (2) configuring LoRA hyperparameters (rank, alpha, target modules), and (3) wrapping the model with PEFT to isolate trainable parameters.

Usage

Use this principle after loading a 4-bit quantized model (QLoRA) to add trainable LoRA adapters. The training_mode must be set to "qlora" to ensure IPEX-LLM applies the correct quantization-aware gradient computation.

Theoretical Basis

LoRA decomposes weight updates as low-rank matrices:

$W^{\prime }=W+\mathrm{\Delta }W=W+BA$

Where:

• $W\in {\mathbb{ℝ}}^{d\times k}$ is the frozen pretrained weight (4-bit quantized)
• $B\in {\mathbb{ℝ}}^{d\times r}$ and $A\in {\mathbb{ℝ}}^{r\times k}$ are trainable (bf16)
• $r\ll \min (d,k)$ is the rank (typically 8 or 16)
• Scaling factor: $\alpha /r$

Related Pages

Implemented By

• Implementation:Intel_Ipex_llm_Get_Peft_Model_QLoRA

Uses Heuristic

• Heuristic:Intel_Ipex_llm_LoRA_Target_All_Linear_Layers