Implementation:Axolotl ai cloud Axolotl Load Lora

Knowledge Sources	Axolotl PEFT Library
Domains	Parameter_Efficient_Finetuning, Model_Architecture
Last Updated	2026-02-06 23:00 GMT

Overview

Concrete tool for injecting LoRA adapters into pre-trained language models provided by the Axolotl framework.

Description

The load_lora function creates a LoraConfig from Axolotl's YAML configuration and applies it to the model using HuggingFace PEFT. It supports auto-discovery of target modules (via lora_target_linear), custom module targeting, LoftQ initialization for quantized models, and mixed adapter models (PeftMixedModel) for combining multiple adapter types. The function handles both fresh LoRA injection for new training and loading existing LoRA checkpoints for continued training.

Usage

Import this function after model loading and before training setup. It transforms a standard PreTrainedModel into a PeftModel with trainable LoRA adapters.

Code Reference

Source Location

• Repository: axolotl
• File: src/axolotl/loaders/adapter.py
• Lines: L70-178

Signature

def load_lora(
    model: PreTrainedModel,
    cfg: DictDefault,
    inference: bool = False,
    config_only: bool = False,
) -> tuple[PreTrainedModel | PeftModel | PeftMixedModel | None, PeftConfig | None]:
    """Inject LoRA adapters into a pre-trained model.

    Args:
        model: Base model to inject adapters into.
        cfg: Configuration with LoRA parameters (lora_r, lora_alpha, etc.).
        inference: Whether loading for inference (loads from checkpoint).
        config_only: Return only the LoraConfig without applying to model.

    Returns:
        Tuple of (PeftModel with LoRA adapters, LoraConfig).
    """

Import

from axolotl.loaders.adapter import load_lora

I/O Contract

Inputs

Name	Type	Required	Description
model	PreTrainedModel	Yes	Base model (optionally quantized) to add LoRA adapters to
cfg	DictDefault	Yes	Config with lora_r, lora_alpha, lora_dropout, lora_target_modules, lora_target_linear, lora_modules_to_save, etc.
inference	bool	No (default: False)	Load existing adapter from checkpoint for inference
config_only	bool	No (default: False)	Return only LoraConfig without wrapping model

Outputs

Name	Type	Description
model	PeftModel or PeftMixedModel or None	Model wrapped with LoRA adapters (None if config_only=True)
peft_config	PeftConfig or None	The LoRA configuration used

Usage Examples

Basic LoRA Injection

from axolotl.loaders.adapter import load_lora
from axolotl.loaders.model import ModelLoader

# Load base model
loader = ModelLoader(cfg, tokenizer)
model, _ = loader.load()

# Inject LoRA adapters
# cfg.lora_r = 16
# cfg.lora_alpha = 32
# cfg.lora_dropout = 0.05
# cfg.lora_target_linear = True
model, lora_config = load_lora(model, cfg)

# Check trainable parameters
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"Trainable: {trainable:,} / {total:,} ({100*trainable/total:.2f}%)")

LoRA with Specific Target Modules

# cfg.lora_target_modules = ["q_proj", "v_proj", "k_proj", "o_proj"]
# cfg.lora_modules_to_save = ["embed_tokens", "lm_head"]
model, lora_config = load_lora(model, cfg)

Related Pages

Implements Principle

• Principle:Axolotl_ai_cloud_Axolotl_LoRA_Adapter_Injection

Requires Environment

• Environment:Axolotl_ai_cloud_Axolotl_CUDA_GPU