Principle:Microsoft LoRA LoRA Weight Extraction

Overview

LoRA Weight Extraction describes the process of extracting LoRA-specific weights from full model checkpoints saved by the HuggingFace Trainer, and converting the naming convention from the modified Transformers fork format to the standard loralib format. This post-training step is necessary because the HuggingFace Trainer saves complete model state dictionaries by default, not just the trainable LoRA parameters.

Why Extraction Is Needed

When training completes, the HuggingFace Trainer calls trainer.save_model(), which invokes model.save_pretrained(). This saves all model parameters to a pytorch_model.bin file, including:

• Frozen pretrained backbone weights -- RoBERTa or DeBERTa V2 parameters (~125M to ~1.5B parameters)
• LoRA matrices -- The small low-rank adaptation matrices (lora_A, lora_B per adapted layer, totaling ~0.3M to ~4.7M parameters)
• Classifier head -- The task-specific classification layer weights

For deployment and distribution, only the LoRA matrices and classifier head are needed. The pretrained backbone weights are already available from the HuggingFace Hub. Extracting just the LoRA weights reduces checkpoint size from hundreds of megabytes (or gigabytes for XXL models) to a few megabytes.

Two-Step Process

The microsoft/LoRA repository provides two utility scripts that perform the extraction:

Step 1: split_lora.py

The split_lora.py script (22 lines) filters the full checkpoint to retain only LoRA-related parameters:

• Parameters whose names contain 'lora' (the LoRA matrices)
• Parameters whose names do not start with 'deberta' or 'roberta' (the classifier head, which is not part of the pretrained backbone)

This filtering logic mirrors the parameter freezing logic in run_glue.py, which freezes all parameters starting with 'deberta' or 'roberta' except those containing 'lora'.

Step 2: convert.py

The convert.py script (21 lines) renames parameter keys from the modified Transformers fork naming convention to the standard loralib naming convention.

The modified fork uses a naming scheme where LoRA matrices are stored as separate sub-modules:

self.query_lora_a.weight    self.query_lora_b.weight
self.value_lora_a.weight    self.value_lora_b.weight

The standard loralib naming uses dot-separated attributes on the parent module:

self.query_proj.lora_A      self.query_proj.lora_B
self.value_proj.lora_A      self.value_proj.lora_B

This conversion is necessary for compatibility with the lora_path loading mechanism in run_glue.py, which expects the loralib-compatible naming.

Naming Convention Differences

Component	Modified Fork Name	Standard loralib Name
Query LoRA down-projection	self.query_lora_a.weight	self.query_proj.lora_A
Query LoRA up-projection	self.query_lora_b.weight	self.query_proj.lora_B
Value LoRA down-projection	self.value_lora_a.weight	self.value_proj.lora_A
Value LoRA up-projection	self.value_lora_b.weight	self.value_proj.lora_B

Note that in loralib convention, lora_A is the down-projection (dimension r x d) and lora_B is the up-projection (dimension d x r), matching the LoRA paper's notation where the effective update is B @ A.

Practical Workflow

The typical post-training workflow is:

• Train the model using run_glue.py with --do_train
• The Trainer saves pytorch_model.bin in the output directory
• Run split_lora.py to extract LoRA + classifier weights
• Run convert.py to rename keys to loralib format
• Use the resulting file with --lora_path for evaluation or further fine-tuning

The extracted LoRA file is typically only a few megabytes, making it easy to distribute, version, and deploy alongside the publicly available base model.

Metadata

Field	Value
Source	Repo (microsoft/LoRA)
Domains	Utilities, NLU, LoRA
Related	Implementation:Microsoft_LoRA_Split_LoRA_Weights