Implementation:Huggingface Peft GraloraModel

Knowledge Sources	Huggingface_Peft
Domains	Deep_Learning, Parameter_Efficient_Finetuning
Last Updated	2026-02-07 14:00 GMT

Overview

Concrete tool for applying GraLoRA (Gradient Low-Rank Adaptation) to pretrained transformer models, provided by the Huggingface PEFT library.

Description

GraloraModel is a tuner class that creates a GraLoRA model from a pretrained transformers model. It uses a variant of random matrix adaptation to inject low-rank adapters with gradient-aware scaling into target Linear and Conv1D modules. The class supports hybrid rank configurations and dropout parameters.

Usage

GraloraModel is typically created internally by calling get_peft_model with a GraloraConfig. It can also be instantiated directly by passing a base model, a GraloraConfig, and an adapter name.

Code Reference

Source Location

• Repository: Huggingface_Peft
• File: src/peft/tuners/gralora/model.py
• Lines: 30-143

Signature

class GraloraModel(BaseTuner):
    prefix: str = "gralora_"
    # Inherits __init__ from BaseTuner:
    # def __init__(self, model, config, adapter_name):
    #     ...

Import

from peft.tuners.gralora import GraloraModel

I/O Contract

Inputs

Name	Type	Required	Description
model	nn.Module	Yes	The pretrained model to adapt
config	GraloraConfig	Yes	Configuration for the GraLoRA adapter (r, alpha, gralora_dropout, gralora_k, hybrid_r)
adapter_name	str	Yes	Name identifier for the adapter, defaults to "default"

Outputs

Name	Type	Description
adapted_model	GraloraModel	Model with GraLoRA adapter layers injected into target modules

Usage Examples

Basic Usage

from peft import get_peft_model, GraloraConfig
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m")
config = GraloraConfig(r=128, target_modules=["q_proj", "v_proj"])
model = get_peft_model(model, config)

Related Pages

• Environment:Huggingface_Peft_Python_Core_Dependencies