Implementation:Intel Ipex llm LISA Finetuning

Knowledge Sources	Intel IPEX-LLM
Domains	Finetuning, LISA, Memory_Efficient_Training
Last Updated	2026-02-09 04:00 GMT

Overview

Concrete tool for LISA (Layer-wise Integrated Sensitivity-based Adaptation) fine-tuning with dynamic layer activation provided by IPEX-LLM.

Description

The train() function implements LISA fine-tuning, which dynamically activates and trains different model layers during training to reduce memory requirements. It uses IPEX-LLM's DynamicLayerActivationCallback to control which layers are trainable at each step. The approach achieves similar performance to full fine-tuning while using significantly less memory by only training a subset of layers at any given time.

Usage

Use this when full fine-tuning is desired but GPU memory is limited. LISA provides a middle ground between LoRA (which restricts parameter updates to low-rank adapters) and full fine-tuning (which updates all parameters).

Code Reference

Source Location

• Repository: Intel IPEX-LLM
• File: python/llm/example/GPU/LLM-Finetuning/LISA/lisa_finetuning.py
• Lines: 1-167

Signature

def train(
    base_model: str = "meta-llama/Llama-2-7b-hf",
    data_path: str = "yahma/alpaca-cleaned",
    output_dir: str = "./bigdl-lisa-alpaca",
    batch_size: int = 128,
    micro_batch_size: int = 8,
    num_epochs: int = 3,
    learning_rate: float = 2e-5,
    cutoff_len: int = 256,
    lisa_activated_layers: int = 1,
    lisa_interval_steps: int = 20,
    gradient_checkpointing: bool = False,
    deepspeed: str = None,
):
    """LISA fine-tuning with dynamic layer activation."""

Import

from ipex_llm.transformers import AutoModelForCausalLM
from ipex_llm.transformers.lisa import DynamicLayerActivationCallback
from transformers import Trainer, TrainingArguments

I/O Contract

Inputs

Name	Type	Required	Description
base_model	str	Yes	HuggingFace model ID or local path
data_path	str	No	Dataset path (default: yahma/alpaca-cleaned)
lisa_activated_layers	int	No	Number of layers active at each step (default: 1)
lisa_interval_steps	int	No	Steps between layer rotation (default: 20)

Outputs

Name	Type	Description
Fine-tuned model	Files	Saved to output_dir
Training metrics	Console	Loss and training progress

Usage Examples

LISA Fine-tuning

python lisa_finetuning.py \
    --base_model "meta-llama/Llama-2-7b-hf" \
    --data_path "yahma/alpaca-cleaned" \
    --lisa_activated_layers 1 \
    --lisa_interval_steps 20 \
    --learning_rate 2e-5

Related Pages

• Environment:Intel_Ipex_llm_XPU_Finetuning_Environment