Environment:Huggingface Transformers PEFT Adapter Env

Knowledge Sources	Huggingface Transformers setup.py
Domains	PEFT, Fine_Tuning, Infrastructure
Last Updated	2026-02-13 20:00 GMT

Overview

Environment with the PEFT (Parameter-Efficient Fine-Tuning) library for LoRA adapter injection, training, saving, and loading.

Description

The PEFT environment enables parameter-efficient fine-tuning methods (LoRA, QLoRA, AdaLoRA, etc.) that modify only a small fraction of model parameters. The Transformers library has deep integration with PEFT through the integrations/peft.py module, providing native model.add_adapter(), model.load_adapter(), and model.save_pretrained() support. This environment requires the peft library (>= 0.18.0) alongside PyTorch and Transformers.

Usage

Required for the PEFT Adapter Integration workflow and the QLoRA Fine-Tuning step of the Model Quantization workflow. Use when you need to fine-tune large models with limited GPU memory by training only adapter weights.

System Requirements

Category	Requirement	Notes
OS	Linux, macOS, Windows	Cross-platform support
Hardware	GPU recommended	CPU possible but very slow for training
VRAM	>= 6GB for 7B model with LoRA	Much less than full fine-tuning

Dependencies

Python Packages

torch >= 2.4.0
peft >= 0.18.0
accelerate >= 1.1.0
transformers >= 5.0
bitsandbytes >= 0.46.1 (optional, for QLoRA)

Credentials

HF_TOKEN: HuggingFace API token (for loading/pushing gated models and adapters).

Quick Install

# For LoRA fine-tuning
pip install transformers[torch] peft>=0.18.0 accelerate

# For QLoRA (4-bit + LoRA)
pip install transformers[torch] peft>=0.18.0 bitsandbytes>=0.46.1 accelerate

Code Evidence

PEFT availability check from src/transformers/utils/import_utils.py:723-724:

@lru_cache
def is_peft_available() -> bool:
    return _is_package_available("peft")

PEFT version requirement from setup.py:109:

"peft>=0.18.0",

Common Errors

Error Message	Cause	Solution
`ImportError: peft is not installed`	PEFT library missing	`pip install peft>=0.18.0`
`ValueError: Adapter not found`	Adapter name does not exist on model	Check adapter name with `model.peft_config`
`RuntimeError: Sizes of tensors must match`	Adapter rank/target mismatch	Verify LoraConfig target_modules match model architecture

Compatibility Notes

Multi-adapter: Transformers supports loading, switching, and combining multiple adapters on a single model.
PEFT + Quantization: QLoRA requires both PEFT and bitsandbytes environments.
Saving: model.save_pretrained() automatically saves only adapter weights when PEFT is active.

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment