Implementation:Huggingface Transformers LoraConfig

Knowledge Sources	PEFT PEFT Docs Transformers Docs
Domains	Parameter_Efficient_Fine_Tuning, NLP, Low_Rank_Adaptation
Last Updated	2026-02-13 00:00 GMT

Overview

Concrete tool for defining the hyperparameters of a Low-Rank Adaptation (LoRA) module, provided by the PEFT library and consumed by Transformers' adapter integration.

Description

LoraConfig is a configuration dataclass from the PEFT library that encapsulates all hyperparameters needed to create and inject LoRA adapter layers into a pretrained model. While it is defined externally in the peft package, it is the primary configuration object accepted by Transformers' model.add_adapter() and model.load_adapter() methods.

The configuration object controls:

• The rank and scaling of the low-rank decomposition matrices
• Which modules in the model receive LoRA layers
• Regularization via dropout
• Bias training behavior
• Weight initialization strategy
• Inference mode flag to disable dropout and gradient computation

When passed to model.add_adapter(adapter_config), Transformers uses PEFT's inject_adapter_in_model function to traverse the model graph, identify target modules, and wrap them with LoRA layers parameterized by this configuration.

Usage

Use LoraConfig whenever you need to:

• Define adapter hyperparameters before injecting adapters via model.add_adapter()
• Experiment with different rank/alpha combinations for LoRA training
• Specify which model layers should receive adapter modules
• Save and reload adapter configurations alongside adapter weights

Code Reference

Source Location

• Repository: peft
• File: src/peft/tuners/lora/config.py (external library)

Signature

class LoraConfig(PeftConfig):
    r: int = 8
    lora_alpha: int = 16
    target_modules: Optional[Union[list[str], str]] = None
    lora_dropout: float = 0.0
    bias: str = "none"
    task_type: Optional[Union[str, TaskType]] = None
    init_lora_weights: bool = True
    modules_to_save: Optional[list[str]] = None
    inference_mode: bool = False

Import

from peft import LoraConfig

I/O Contract

Inputs

Name	Type	Required	Description
r	int	No	Rank of the low-rank decomposition matrices. Higher values increase expressiveness and parameter count. Default: 8.
lora_alpha	int	No	Scaling factor for the LoRA update. The effective scaling is lora_alpha / r. Default: 16.
target_modules	list[str] or str or None	No	Names of modules to apply LoRA to (e.g., ["q_proj", "v_proj"]). If None, PEFT auto-detects all linear layers. Default: None.
lora_dropout	float	No	Dropout probability applied to the LoRA layers during training. Default: 0.0.
bias	str	No	Bias training strategy. One of "none", "all", or "lora_only". Default: "none".
task_type	str or TaskType or None	No	The task type for the model (e.g., "CAUSAL_LM", "SEQ_2_SEQ_LM"). Affects output layer handling. Default: None.
init_lora_weights	bool	No	Whether to initialize LoRA weights with the default scheme (A: Kaiming, B: zeros). Set to False for random init. Default: True.
modules_to_save	list[str] or None	No	Additional modules to make trainable and save alongside the adapter (e.g., ["embed_tokens", "lm_head"]). Default: None.
inference_mode	bool	No	Whether the adapter is for inference only (disables dropout, disables gradients). Default: False.

Outputs

Name	Type	Description
config	LoraConfig	A configuration object that can be passed to model.add_adapter() or serialized/deserialized alongside adapter weights.

Usage Examples

Basic Usage: Standard LoRA Configuration

from peft import LoraConfig

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

QLoRA Configuration (with All Linear Layers)

from peft import LoraConfig

# target_modules=None triggers auto-detection of all linear layers
qlora_config = LoraConfig(
    r=64,
    lora_alpha=16,
    target_modules=None,
    lora_dropout=0.1,
    bias="none",
    task_type="CAUSAL_LM",
)

Configuration with Saved Modules

from peft import LoraConfig

config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    modules_to_save=["embed_tokens", "lm_head"],
    task_type="CAUSAL_LM",
)

Related Pages

Implements Principle

• Principle:Huggingface_Transformers_LoRA_Configuration

Requires Environment

• Environment:Huggingface_Transformers_PEFT_Adapter_Env