Implementation:Huggingface Peft LoraConfig

Field	Value
Sources	Repo: PEFT, Doc: PEFT Docs
Domains	NLP, Parameter_Efficient_Finetuning
Last Updated	2026-02-07 00:00 GMT

Overview

Concrete tool for configuring Low-Rank Adaptation parameters provided by the PEFT library.

Description

LoraConfig is the central configuration dataclass in the Hugging Face PEFT library that encapsulates all hyperparameters and behavioral options for Low-Rank Adaptation (LoRA). It extends the base PeftConfig class and serves as the single entry point through which users specify how LoRA should be applied to a pretrained model.

The class controls every aspect of the LoRA adaptation process:

Rank and scaling — the low-rank dimension r and the scaling factor lora_alpha that together determine the effective magnitude of the adaptation.
Target selection — which model modules receive LoRA adapters (target_modules), which layers to transform (layers_to_transform), and which modules to exclude (exclude_modules).
Initialization strategy — how the low-rank matrices are initialized, supporting standard, Gaussian, EVA, OLoRA, PiSSA, CorDA, LoftQ, and orthogonal initialization schemes via init_lora_weights.
Variants — enabling RSLoRA scaling (use_rslora), DoRA weight decomposition (use_dora), and bias training configurations.

When passed to get_peft_model(), the LoraConfig instance drives the injection of LoRA layers into the specified modules of the frozen base model, creating a parameter-efficient trainable wrapper.

Usage

Instantiate LoraConfig when:

Preparing a model for LoRA fine-tuning — create a config specifying rank, target modules, and scaling, then pass it to get_peft_model() to wrap the base model.
Setting up QLoRA training — combine LoraConfig with a 4-bit quantized base model for memory-efficient fine-tuning on consumer hardware.
Enabling DoRA — set use_dora=True to apply the weight-decomposed variant of LoRA for improved fine-tuning fidelity.
Customizing per-layer ranks — use rank_pattern and alpha_pattern dictionaries to assign different ranks to different layers based on their importance.
Saving and loading adapters — LoraConfig is serialized alongside the adapter weights, enabling reproducible loading of trained adapters.

Code Reference

Source Location

Property	Value
Repository	huggingface/peft
File	`src/peft/tuners/lora/config.py`
Lines	L322-889

Signature

@dataclass
class LoraConfig(PeftConfig):
    r: int = 8  # LoRA rank
    target_modules: Optional[Union[list[str], str]] = None
    exclude_modules: Optional[Union[list[str], str]] = None
    lora_alpha: int = 8
    lora_dropout: float = 0.0
    fan_in_fan_out: bool = False
    bias: Literal["none", "all", "lora_only"] = "none"
    use_rslora: bool = False
    modules_to_save: Optional[list[str]] = None
    init_lora_weights: bool | Literal[
        "gaussian", "eva", "olora", "pissa",
        "pissa_niter_[number of iters]", "corda",
        "loftq", "orthogonal"
    ] = True
    layers_to_transform: Optional[Union[list[int], int]] = None
    layers_pattern: Optional[Union[list[str], str]] = None
    rank_pattern: Optional[dict] = None
    alpha_pattern: Optional[dict] = None
    use_dora: bool = False
    lora_bias: bool = False

Import Statement

from peft import LoraConfig

I/O Contract

Inputs

Parameter	Type	Required	Default	Description
r	`int`	No	`8`	LoRA rank — the dimension of the low-rank decomposition matrices. Controls the expressiveness vs. efficiency trade-off.
target_modules	`Optional[Union[list[str], str]]`	No	`None`	Names of the modules to apply LoRA to. If None, PEFT automatically selects modules based on the model architecture. Can be a list of module names or a single regex string.
exclude_modules	`Optional[Union[list[str], str]]`	No	`None`	Names of modules to explicitly exclude from LoRA adaptation, even if matched by target_modules.
lora_alpha	`int`	No	`8`	The scaling factor for LoRA. The effective scaling applied to the low-rank update is lora_alpha / r (or lora_alpha / sqrt(r) when RSLoRA is enabled).
lora_dropout	`float`	No	`0.0`	Dropout probability applied to the LoRA layers during training. Set to 0.0 to disable dropout.
fan_in_fan_out	`bool`	No	`False`	Set to True if the layer stores weights as (fan_in, fan_out), as in some Conv1D implementations (e.g., GPT-2).
bias	`Literal["none", "all", "lora_only"]`	No	`"none"`	Controls which bias parameters are trained. "none" trains no biases, "all" trains all biases in the model, "lora_only" trains only biases of LoRA-adapted modules.
use_rslora	`bool`	No	`False`	When True, uses Rank-Stabilized LoRA scaling (lora_alpha / sqrt(r)) instead of the standard (lora_alpha / r).
modules_to_save	`Optional[list[str]]`	No	`None`	List of module names (besides LoRA layers) that should be set as trainable and saved in the final checkpoint (e.g., classification heads).
init_lora_weights	Literal["gaussian", "eva", "olora", "pissa", "pissa_niter_[number of iters]", "corda", "loftq", "orthogonal"]	No	`True`	Initialization strategy for LoRA weights. True uses Kaiming uniform for A and zeros for B. False skips initialization. Named strategies provide specialized initialization methods.
layers_to_transform	`Optional[Union[list[int], int]]`	No	`None`	Indices of specific layers to apply LoRA to. If None, LoRA is applied to all matching layers.
layers_pattern	`Optional[Union[list[str], str]]`	No	`None`	Pattern used to match layer names when using layers_to_transform. Allows mapping layer indices to specific named layers in the model.
rank_pattern	`Optional[dict]`	No	`None`	Dictionary mapping module names to custom rank values, enabling per-layer rank customization.
alpha_pattern	`Optional[dict]`	No	`None`	Dictionary mapping module names to custom alpha values, enabling per-layer scaling customization.
use_dora	`bool`	No	`False`	When True, enables Weight-Decomposed Low-Rank Adaptation (DoRA), which decomposes weight updates into magnitude and direction components.
lora_bias	`bool`	No	`False`	When True, adds a trainable bias term to the LoRA layers.

Outputs

Output	Type	Description
config	`LoraConfig`	A fully initialized configuration object that can be passed to get_peft_model() to apply LoRA adapters to a pretrained model. Serializable for saving/loading alongside adapter weights.

Usage Examples

Basic LoRA Configuration

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

# Load the base model
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

# Create a basic LoRA configuration
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
)

# Apply LoRA to the model
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.0622

QLoRA Configuration with 4-bit Quantization

import torch
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

# Configure 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)

# Create LoRA config for QLoRA
qlora_config = LoraConfig(
    r=64,
    lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.1,
    bias="none",
)

# Apply LoRA adapters on top of the quantized model
model = get_peft_model(model, qlora_config)
model.print_trainable_parameters()

DoRA Configuration

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

# Enable DoRA for weight-decomposed adaptation
dora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    use_dora=True,
    lora_dropout=0.05,
    bias="none",
)

model = get_peft_model(model, dora_config)
model.print_trainable_parameters()

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment