Implementation:Huggingface Transformers Add Adapter

Knowledge Sources	Transformers PEFT Docs Transformers Docs
Domains	Parameter_Efficient_Fine_Tuning, NLP, Model_Architecture
Last Updated	2026-02-13 00:00 GMT

Overview

Concrete tool for injecting a fresh PEFT adapter into a pretrained model for training, provided by the PeftAdapterMixin class in Hugging Face Transformers.

Description

model.add_adapter() is a method on PeftAdapterMixin, which is mixed into all Transformers PreTrainedModel subclasses. It accepts a PEFT configuration object (e.g., LoraConfig) and injects a new set of trainable adapter parameters into the model.

The method performs the following operations:

1. Version check: Validates that the installed PEFT version meets the minimum requirement (>= 0.18.0)
2. Name management: Uses the provided adapter_name (defaulting to "default") and raises an error if an adapter with that name already exists
3. Type validation: Ensures adapter_config is an instance of PeftConfig
4. Base model path recording: Sets adapter_config.base_model_name_or_path from the model's name_or_path attribute for later serialization
5. Injection: Calls PEFT's inject_adapter_in_model(adapter_config, self, adapter_name) to traverse the model, identify target modules, and wrap them with adapter layers
6. Activation: Calls self.set_adapter(adapter_name) to immediately activate the newly injected adapter

After this method completes, the model has trainable adapter parameters and frozen base parameters. Only the adapter parameters will receive gradients during .backward().

Usage

Use add_adapter when you want to:

• Inject a new LoRA, IA3, or other PEFT adapter into a loaded base model
• Add additional adapters to a model that already has adapters (for multi-task setups)
• Prepare a model for training where only adapter parameters should be updated

Code Reference

Source Location

• Repository: transformers
• File: src/transformers/integrations/peft.py (lines 636-674)

Signature

def add_adapter(
    self,
    adapter_config,          # PeftConfig (e.g., LoraConfig)
    adapter_name: str | None = None,  # defaults to "default"
) -> None

Import

# add_adapter is a method on PreTrainedModel via PeftAdapterMixin
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("model-name")
model.add_adapter(config)  # method call

I/O Contract

Inputs

Name	Type	Required	Description
adapter_config	peft.PeftConfig	Yes	The configuration of the adapter to add. Supported types include LoraConfig, IA3Config, and other non-prompt-learning PEFT configs.
adapter_name	str or None	No	The name to assign to this adapter. Defaults to "default". Must be unique among existing adapters on the model.

Outputs

Name	Type	Description
(none)	None	The method modifies the model in-place. After execution, adapter layers are injected and activated. The model's peft_config dict contains the new adapter configuration.

Usage Examples

Basic Usage: Add a LoRA Adapter

from transformers import AutoModelForCausalLM
from peft import LoraConfig

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    device_map="auto",
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)

model.add_adapter(lora_config)
# Model now has trainable LoRA parameters; base weights are frozen

Multi-Adapter Setup

from transformers import AutoModelForCausalLM
from peft import LoraConfig

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

# First adapter for summarization
summarization_config = LoraConfig(r=8, target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM")
model.add_adapter(summarization_config, adapter_name="summarization")

# Second adapter for translation
translation_config = LoraConfig(r=16, target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM")
model.add_adapter(translation_config, adapter_name="translation")

# Switch between adapters
model.set_adapter("summarization")
# ... run inference for summarization

model.set_adapter("translation")
# ... run inference for translation

Related Pages

Implements Principle

• Principle:Huggingface_Transformers_Adapter_Injection

Requires Environment

• Environment:Huggingface_Transformers_PEFT_Adapter_Env