Implementation:Huggingface Peft Get Peft Model

Overview

Concrete tool for injecting PEFT adapter layers into a pretrained model provided by the PEFT library. The get_peft_model function is the primary entry point for converting a standard pretrained model into a parameter-efficient fine-tuning model by wrapping it with the appropriate PeftModel subclass based on the adapter configuration.

Description

get_peft_model is a factory function that takes a pretrained model and a PEFT configuration object, then returns the model wrapped in the appropriate PeftModel (or PeftMixedModel) subclass. The function performs the following operations:

Extracts model configuration -- Retrieves the model config and updates the base_model_name_or_path on the PEFT config.
Validates existing adapters -- Checks whether the model already has PEFT adapter layers injected and warns the user if a second modification is attempted.
Handles revision tracking -- Propagates the revision parameter to the PEFT config for reproducible model loading.
Routes to the correct PeftModel subclass -- Uses MODEL_TYPE_TO_PEFT_MODEL_MAPPING to select the task-specific wrapper class (e.g., PeftModelForCausalLM for causal language modeling). If no task_type is set (and the config is not prompt learning), the base PeftModel class is used.
Supports mixed adapters -- When mixed=True, returns a PeftMixedModel that allows combining different adapter types on the same base model.

The underlying PeftModel constructor delegates to the appropriate tuner class (e.g., LoraModel) via PEFT_TYPE_TO_TUNER_MAPPING, which performs the actual layer replacement and parameter freezing.

Usage

Call get_peft_model after creating a PEFT configuration and before starting training. The returned model is ready for parameter-efficient fine-tuning with any standard training loop or the HuggingFace Trainer.

Code Reference

Source file: src/peft/mapping_func.py, lines 30--128

Import:

from peft import get_peft_model

Full signature:

def get_peft_model(
    model: PreTrainedModel,
    peft_config: PeftConfig,
    adapter_name: str = "default",
    mixed: bool = False,
    autocast_adapter_dtype: bool = True,
    revision: Optional[str] = None,
    low_cpu_mem_usage: bool = False,
) -> PeftModel | PeftMixedModel:

I/O Contract

Inputs

Parameter	Type	Default	Description
`model`	`PreTrainedModel`	(required)	The base transformer model to be wrapped with PEFT adapters. This model is modified in-place.
`peft_config`	`PeftConfig`	(required)	Configuration object specifying the PEFT method, target modules, hyperparameters, and task type.
`adapter_name`	`str`	`"default"`	Name identifier for the injected adapter. Used when managing multiple adapters on the same model.
`mixed`	`bool`	`False`	If `True`, returns a `PeftMixedModel` that supports combining different compatible adapter types.
`autocast_adapter_dtype`	`bool`	`True`	Whether to cast adapter weights from float16/bfloat16 to float32 for stable training. Only affects select PEFT tuners.
`revision`	`Optional[str]`	`None`	Revision of the base model to record in the PEFT config. Used for reproducibility when saving/loading.
`low_cpu_mem_usage`	`bool`	`False`	Creates empty adapter weights on meta device. Speeds up loading but should not be used if training from scratch.

Output

Type	Description
`PeftModel` or `PeftMixedModel`	The input model wrapped with PEFT adapter layers. The specific `PeftModel` subclass depends on the `task_type` in the config: `PeftModelForCausalLM`, `PeftModelForSeq2SeqLM`, `PeftModelForSequenceClassification`, `PeftModelForTokenClassification`, `PeftModelForQuestionAnswering`, `PeftModelForFeatureExtraction`, or the base `PeftModel`.

Usage Examples

Basic Usage with LoraConfig

from transformers import AutoModelForCausalLM
from peft import get_peft_model, LoraConfig

# Load a pretrained model
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

# Define LoRA configuration
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
)

# Inject adapter layers
model = get_peft_model(model, lora_config)

# Verify trainable parameters
model.print_trainable_parameters()
# Output: trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.0622

Usage with task_type for Sequence Classification

from transformers import AutoModelForSequenceClassification
from peft import get_peft_model, LoraConfig, TaskType

model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-uncased", num_labels=2
)

config = LoraConfig(
    task_type=TaskType.SEQ_CLS,
    r=8,
    lora_alpha=16,
    target_modules=["query", "value"],
)

# Returns a PeftModelForSequenceClassification instance
model = get_peft_model(model, config)

Usage with Mixed Adapter Types

from transformers import AutoModelForCausalLM
from peft import get_peft_model, LoraConfig

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
)

# Enable mixed mode for combining different adapter types later
model = get_peft_model(model, lora_config, mixed=True)

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment