Implementation:Huggingface Peft GraloraConfig

Knowledge Sources	Huggingface_Peft
Domains	Deep_Learning, Parameter_Efficient_Finetuning
Last Updated	2026-02-07 14:00 GMT

Overview

Concrete configuration dataclass for the GraLoRA PEFT adapter method provided by the Huggingface PEFT library.

Description

GraloraConfig controls the hyperparameters for the GraLoRA (Gradient-based Low-Rank Adaptation) adapter, which uses subblock decomposition to increase expressivity while preserving the same total parameter count as LoRA. The gralora_k parameter splits the adapter into subblocks, multiplying expressivity by that factor. This class extends PeftConfig and also supports a Hybrid GraLoRA mode combining GraLoRA with vanilla LoRA.

Usage

Import and use GraloraConfig when you want to apply GraLoRA fine-tuning, which provides higher expressivity than LoRA at the same parameter count by decomposing the adapter into multiple subblocks. Use hybrid_r > 0 to combine GraLoRA with vanilla LoRA.

Code Reference

Source Location

• Repository: Huggingface_Peft
• File: src/peft/tuners/gralora/config.py
• Lines: 22-183

Signature

@dataclass
class GraloraConfig(PeftConfig):
    r: int = 32
    hybrid_r: int = 0
    target_modules: Optional[Union[list[str], str]] = None
    alpha: int = 64
    gralora_dropout: float = 0.0
    gralora_k: int = 2
    fan_in_fan_out: bool = False
    bias: str = "none"
    modules_to_save: Optional[list[str]] = None
    init_weights: bool = True
    layers_to_transform: Optional[Union[list[int], int]] = None
    layers_pattern: Optional[str] = None

Import

from peft import GraloraConfig

I/O Contract

Inputs

Name	Type	Default	Description
r	int	32	GraLoRA rank. Must be divisible by gralora_k. Total parameter count matches LoRA with same rank.
hybrid_r	int	0	Rank allocated to vanilla LoRA in Hybrid GraLoRA mode. Active when > 0.
target_modules	Optional[Union[list[str], str]]	None	Module names or regex to apply adapter to. Supports 'all-linear'.
alpha	int	64	Scaling factor. Scale becomes alpha / (r + hybrid_r).
gralora_dropout	float	0.0	Dropout probability for the GraLoRA adapter.
gralora_k	int	2	Number of subblocks. r must be divisible by gralora_k. Recommended: 2 for rank <= 32, 4 for rank >= 64.
fan_in_fan_out	bool	False	Set True if layer stores weight as (fan_in, fan_out).
bias	str	"none"	Bias type. Can be 'none', 'all', or 'gralora_only'.
modules_to_save	Optional[list[str]]	None	Additional modules to set as trainable and save in the final checkpoint.
init_weights	bool	True	Whether to initialize GraLoRA layer weights with default initialization.
layers_to_transform	Optional[Union[list[int], int]]	None	Specific layer indexes to transform.
layers_pattern	Optional[str]	None	Layer pattern name when layers_to_transform is set.

Outputs

Name	Type	Description
instance	GraloraConfig	Configuration object to pass to get_peft_model

Usage Examples

Basic Configuration

from peft import GraloraConfig, get_peft_model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("model-name")
config = GraloraConfig(
    r=32,
    gralora_k=2,
    alpha=64,
    target_modules=["q_proj", "v_proj"],
)
model = get_peft_model(model, config)

Related Pages

• Environment:Huggingface_Peft_Python_Core_Dependencies