Implementation:Huggingface Peft DeloraConfig

Knowledge Sources	Huggingface_Peft
Domains	Deep_Learning, Parameter_Efficient_Finetuning
Last Updated	2026-02-07 14:00 GMT

Overview

Concrete configuration dataclass for the DeLoRA PEFT adapter method provided by the Huggingface PEFT library.

Description

DeloraConfig controls the hyperparameters for the DeLoRA (Decoupled Low-Rank Adaptation) adapter, which applies low-rank weight updates with an explicit upper bound on the Frobenius norm of weight changes via the delora_lambda parameter. This prevents the finetuned model from deviating too far from the original. This class extends PeftConfig and supports per-layer rank and lambda patterns.

Usage

Import and use DeloraConfig when you want to apply DeLoRA fine-tuning to a pretrained model, particularly when you want to constrain the magnitude of weight updates during adaptation to prevent catastrophic forgetting or excessive model drift.

Code Reference

Source Location

• Repository: Huggingface_Peft
• File: src/peft/tuners/delora/config.py
• Lines: 23-155

Signature

@dataclass
class DeloraConfig(PeftConfig):
    r: int = 8
    delora_lambda: int = 15
    module_dropout: float = 0.0
    target_modules: Optional[Union[list[str], str]] = None
    exclude_modules: Optional[Union[list[str], str]] = None
    bias: str = "none"
    init_weights: bool = True
    layers_to_transform: Optional[Union[list[int], int]] = None
    layers_pattern: Optional[Union[list[str], str]] = None
    rank_pattern: Optional[dict] = field(default_factory=dict)
    lambda_pattern: Optional[dict] = field(default_factory=dict)
    modules_to_save: Optional[list[str]] = None

Import

from peft import DeloraConfig

I/O Contract

Inputs

Name	Type	Default	Description
r	int	8	The rank of the DeLoRA adapter.
delora_lambda	int	15	Initial boundary value setting an upper bound to the Frobenius norm of weight changes.
module_dropout	float	0.0	Dropout probability for disabling DeLoRA modules during training.
target_modules	Optional[Union[list[str], str]]	None	Module names or regex to apply adapter to. Supports 'all-linear'.
exclude_modules	Optional[Union[list[str], str]]	None	Module names or regex to exclude from adapter.
bias	str	"none"	Bias type. Can be 'none', 'all', or 'delora_only'.
init_weights	bool	True	If True, A uses kaiming uniform and B uses zeros. If False, both use kaiming uniform.
layers_to_transform	Optional[Union[list[int], int]]	None	Specific layer indexes to apply transformations to.
layers_pattern	Optional[Union[list[str], str]]	None	Layer pattern name when layers_to_transform is set.
rank_pattern	Optional[dict]	{}	Mapping from layer names or regex to custom ranks per layer.
lambda_pattern	Optional[dict]	{}	Mapping from layer names or regex to custom lambda values per layer.
modules_to_save	Optional[list[str]]	None	Additional modules to set as trainable and save in the final checkpoint.

Outputs

Name	Type	Description
instance	DeloraConfig	Configuration object to pass to get_peft_model

Usage Examples

Basic Configuration

from peft import DeloraConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("model-name")
config = DeloraConfig(
    r=8,
    delora_lambda=15,
    target_modules=["q_proj", "v_proj"],
)
model = get_peft_model(model, config)

Related Pages

• Environment:Huggingface_Peft_Python_Core_Dependencies