Implementation:Intel Ipex llm ReLoRA Finetuning

Knowledge Sources	Intel IPEX-LLM ReLoRA
Domains	Finetuning, ReLoRA, Quantization
Last Updated	2026-02-09 04:00 GMT

Overview

Concrete tool for ReLoRA (Restart Low-Rank Adaptation) fine-tuning with periodic weight merging and reset provided by IPEX-LLM.

Description

The train() function implements the ReLoRA training procedure, which periodically merges LoRA adapter weights into the base model and restarts the adapter training. This approach enables training with higher effective rank while maintaining low memory usage. It uses IPEX-LLM's ReLoRATrainer and supports 4-bit quantization via BitsAndBytesConfig, DeepSpeed integration, gradient checkpointing, and CPU offloading during weight resets.

Usage

Use this when fine-tuning a model with ReLoRA to achieve higher effective rank than standard LoRA without proportionally increasing memory. It is suited for scenarios where standard LoRA rank is insufficient but full fine-tuning is too expensive.

Code Reference

Source Location

Repository: Intel IPEX-LLM
File: python/llm/example/GPU/LLM-Finetuning/ReLora/alpaca_relora_finetuning.py
Lines: 1-302

Signature

def train(
    base_model: str = "meta-llama/Llama-2-7b-hf",
    saved_low_bit_model: str = None,
    data_path: str = "yahma/alpaca-cleaned",
    output_dir: str = "./bigdl-qlora-alpaca",
    bf16: bool = True,
    batch_size: int = 128,
    micro_batch_size: int = 2,
    num_epochs: int = 3,
    learning_rate: float = 3e-5,
    cutoff_len: int = 256,
    lora_r: int = 8,
    lora_alpha: int = 16,
    lora_dropout: float = 0.05,
    lora_target_modules: List[str] = [
        "q_proj", "v_proj", "k_proj", "o_proj",
        "up_proj", "down_proj", "gate_proj"
    ],
    training_mode: str = "relora",
    relora_steps: int = 300,
    relora_warmup_steps: int = 10,
    relora_cpu_offload: bool = True,
    gradient_checkpointing: bool = False,
    deepspeed: str = None,
):
    """ReLoRA fine-tuning with periodic weight restart."""

Import

from ipex_llm.transformers import AutoModelForCausalLM
from ipex_llm.transformers import ReLoRATrainer
from ipex_llm.transformers.qlora import get_peft_model, prepare_model_for_kbit_training, LoraConfig

I/O Contract

Inputs

Name	Type	Required	Description
base_model	str	Yes	HuggingFace model ID or local path
data_path	str	No	Dataset path (default: yahma/alpaca-cleaned)
training_mode	str	No	Must be "relora" for ReLoRA training
relora_steps	int	No	Steps between weight resets (default: 300)
relora_warmup_steps	int	No	LR warmup steps after reset (default: 10)
relora_cpu_offload	bool	No	Offload to CPU during reset (default: True)
saved_low_bit_model	str	No	Path to pre-quantized model for faster loading

Outputs

Name	Type	Description
Model checkpoints	Files	Saved to output_dir at each relora_steps interval
Final merged model	Files	Adapter merged into base after training

Usage Examples

ReLoRA Fine-tuning

python alpaca_relora_finetuning.py \
    --base_model "meta-llama/Llama-2-7b-hf" \
    --data_path "yahma/alpaca-cleaned" \
    --output_dir "./relora-output" \
    --training_mode "relora" \
    --relora_steps 300 \
    --relora_warmup_steps 10 \
    --bf16 True

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment