Implementation:Hiyouga LLaMA Factory Model Adapter
| Knowledge Sources | |
|---|---|
| Domains | Machine Learning, Parameter-Efficient Fine-Tuning |
| Last Updated | 2026-02-06 19:00 GMT |
Overview
Core adapter initialization module that bridges model loading with the chosen fine-tuning strategy in LLaMA-Factory.
Description
The adapter module provides the init_adapter function, which is the central dispatch point for configuring how a pretrained model is fine-tuned. It supports four fine-tuning methods: full-parameter tuning, freeze (partial-parameter) tuning, LoRA (including DoRA and PiSSA variants), and OFT (Orthogonal Fine-Tuning). Built on HuggingFace's PEFT library, it handles complex workflows such as multi-adapter merging, adapter resumption, Unsloth and KTransformers integration, vision-language model component freezing, and trainable parameter upcasting to float32 for quantized training.
Usage
Call init_adapter during model setup, after loading the base model and before starting training. It is invoked by the model loader and returns the model with the appropriate adapter configuration applied.
Code Reference
Source Location
- Repository: Hiyouga_LLaMA_Factory
- File: src/llamafactory/model/adapter.py
- Lines: 1-366
Signature
def init_adapter(
config: "PretrainedConfig",
model: "PreTrainedModel",
model_args: "ModelArguments",
finetuning_args: "FinetuningArguments",
is_trainable: bool,
) -> "PreTrainedModel":
"""Initialize the adapters.
Support full-parameter, freeze and LoRA training.
Note that the trainable parameters must be cast to float32.
"""
def _setup_full_tuning(
model: "PreTrainedModel",
finetuning_args: "FinetuningArguments",
is_trainable: bool,
cast_trainable_params_to_fp32: bool,
) -> None: ...
def _setup_freeze_tuning(
model: "PreTrainedModel",
finetuning_args: "FinetuningArguments",
is_trainable: bool,
cast_trainable_params_to_fp32: bool,
) -> None: ...
def _setup_lora_tuning(
config: "PretrainedConfig",
model: "PreTrainedModel",
model_args: "ModelArguments",
finetuning_args: "FinetuningArguments",
is_trainable: bool,
cast_trainable_params_to_fp32: bool,
) -> "PeftModel": ...
Import
from llamafactory.model.adapter import init_adapter
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| config | PretrainedConfig | Yes | The model's pretrained configuration |
| model | PreTrainedModel | Yes | The loaded pretrained model instance |
| model_args | ModelArguments | Yes | Model configuration including adapter paths and quantization settings |
| finetuning_args | FinetuningArguments | Yes | Fine-tuning configuration specifying method, targets, and hyperparameters |
| is_trainable | bool | Yes | Whether the model is being prepared for training (True) or inference (False) |
Outputs
| Name | Type | Description |
|---|---|---|
| model | PreTrainedModel or PeftModel | The model with adapters initialized, frozen parameters set, and trainable params optionally upcast to fp32 |
Usage Examples
from transformers import AutoModelForCausalLM, AutoConfig
from llamafactory.model.adapter import init_adapter
from llamafactory.hparams import ModelArguments, FinetuningArguments
config = AutoConfig.from_pretrained("meta-llama/Llama-2-7b-hf")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
model_args = ModelArguments(model_name_or_path="meta-llama/Llama-2-7b-hf")
finetuning_args = FinetuningArguments(
finetuning_type="lora",
lora_rank=16,
lora_target="all",
)
# Initialize LoRA adapter for training
model = init_adapter(config, model, model_args, finetuning_args, is_trainable=True)
# For inference (loads and merges existing adapters)
model_args_infer = ModelArguments(
model_name_or_path="meta-llama/Llama-2-7b-hf",
adapter_name_or_path="./lora_checkpoint",
)
model = init_adapter(config, model, model_args_infer, finetuning_args, is_trainable=False)