Principle:Huggingface Transformers Adapter Loading And Switching
| Knowledge Sources | |
|---|---|
| Domains | Parameter_Efficient_Fine_Tuning, NLP, Model_Serving |
| Last Updated | 2026-02-13 00:00 GMT |
Overview
Adapter loading and switching enables dynamic attachment of pre-trained adapter weights to a base model at inference time, and the ability to activate, deactivate, or swap between multiple adapters without reloading the base model.
Description
One of the most powerful features of the PEFT integration in Transformers is the ability to load adapter weights from saved checkpoints and switch between them at runtime. This enables multi-tenant model serving where a single base model in memory serves multiple tasks by swapping lightweight adapters.
The adapter loading and switching workflow involves several complementary operations:
- Loading (
load_adapter): Downloads or reads adapter weights and configuration from a local path or Hub repository, creates the adapter layers in the model, and loads the weights into them. The adapter is registered under a specified name and can be loaded as trainable or frozen for inference. - Activation (
set_adapter): Switches the model to use a specific adapter (or a list of adapters for multi-adapter inference). All adapter layers in the model are instructed to route computation through the specified adapter. - Enabling (
enable_adapters): Re-enables all adapters after they have been disabled. This restores the adapter-augmented computation. - Disabling (
disable_adapters): Temporarily disables all adapters, causing the model to behave as if it were the original base model. The adapter weights remain in memory but are not used in the forward pass. - Hotswapping: A memory-efficient mechanism for replacing adapter weights in-place without allocating new parameters. This is particularly valuable when the model has been compiled with
torch.compile, as hotswapping avoids triggering recompilation.
Key design decisions:
- Lazy injection: When loading an adapter, the adapter layers are injected into the model graph only if they do not already exist (for non-hotswap loads)
- State dict mapping: Adapter weights are loaded using the model's standard
_load_pretrained_modelinfrastructure, with weight conversion mappings for models that have non-standard architectures (e.g., fused MoE experts) - Adapter name scoping: Each adapter is uniquely identified by name, preventing conflicts and enabling precise activation control
Usage
Use adapter loading and switching when you need to:
- Load a pre-trained adapter for inference on a specific task
- Serve multiple tasks from a single base model by switching adapters
- Compare base model vs. adapter-augmented outputs by toggling adapters on/off
- Hotswap adapters in a compiled model without recompilation overhead
- Load an adapter as trainable for continued fine-tuning
Theoretical Basis
Adapter switching exploits the additive structure of PEFT methods. At inference time, the model computes:
y = W * x + (alpha / r) * B_active * A_active * x
where B_active and A_active are the matrices of the currently active adapter. Switching adapters changes which (A, B) pair is used in this computation without modifying W.
When adapters are disabled:
y = W * x
This is exactly the base model's behavior, since the adapter contribution is zeroed out.
For multi-adapter inference (activating multiple adapters simultaneously):
y = W * x + sum_i (alpha_i / r_i) * B_i * A_i * x
This enables adapter composition where the effects of multiple fine-tuning tasks are combined.
Hotswapping is an optimization that avoids the memory allocation cost of creating new adapter layers. Instead of deleting the old adapter and injecting a new one, hotswap directly overwrites the weight tensors:
A_old.data = A_new B_old.data = B_new
When combined with torch.compile, this avoids graph retracing because the tensor shapes and the computation graph remain identical (assuming same rank). For different ranks, enable_peft_hotswap(target_rank=max_rank) pre-allocates buffers at the maximum expected rank, padding smaller adapters with zeros.
The cost of loading an adapter is O(adapter_size) for the file I/O and weight copying, which is negligible compared to loading a full model. This makes adapter switching suitable for real-time applications where task context changes frequently.