Implementation:Huggingface Peft IA3Model
| Knowledge Sources | |
|---|---|
| Domains | Deep_Learning, Parameter_Efficient_Finetuning |
| Last Updated | 2026-02-07 14:00 GMT |
Overview
Concrete tool for applying (IA)^3 (Infused Adapter by Inhibiting and Amplifying Inner Activations) to pretrained transformer models, provided by the Huggingface PEFT library.
Description
IA3Model is a tuner class that creates an (IA)^3 model from a pretrained transformers model. It rescales inner activations with learned vectors, distinguishing between attention and feedforward modules. The method supports Linear, Conv2d, Conv3d, and Conv1D layers, as well as bitsandbytes 8-bit and 4-bit quantized layers. The method is described in https://huggingface.co/papers/2205.05638.
Usage
IA3Model is typically created internally by calling get_peft_model with an IA3Config. It can also be instantiated directly by passing a base model, an IA3Config, and an adapter name. The config requires specifying both target_modules and feedforward_modules.
Code Reference
Source Location
- Repository: Huggingface_Peft
- File: src/peft/tuners/ia3/model.py
- Lines: 36-316
Signature
class IA3Model(BaseTuner):
prefix: str = "ia3_"
# Inherits __init__ from BaseTuner:
# def __init__(self, model, config, adapter_name):
# ...
Import
from peft.tuners.ia3 import IA3Model
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| model | nn.Module | Yes | The pretrained model to adapt |
| config | IA3Config | Yes | Configuration for the (IA)^3 adapter (target_modules, feedforward_modules, fan_in_fan_out, init_ia3_weights) |
| adapter_name | str | Yes | Name identifier for the adapter, defaults to "default" |
Outputs
| Name | Type | Description |
|---|---|---|
| adapted_model | IA3Model | Model with (IA)^3 learned rescaling vectors injected into target modules |
Usage Examples
Basic Usage
from peft import get_peft_model, IA3Config
from transformers import AutoModelForSeq2SeqLM
model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
config = IA3Config(
target_modules=["k", "v", "w0"],
feedforward_modules=["w0"],
)
model = get_peft_model(model, config)