Implementation:Microsoft LoRA Split LoRA Weights

Overview

Split LoRA Weights is an API Doc for the two post-training utility scripts in the microsoft/LoRA repository: split_lora.py (which extracts LoRA-specific parameters from full HuggingFace Trainer checkpoints) and convert.py (which renames parameter keys from the modified Transformers fork naming to standard loralib naming).

Source Files

File	Lines	Description
`examples/NLU/utils/split_lora.py`	1-22	Extracts LoRA + classifier parameters from full checkpoint
`examples/NLU/utils/convert.py`	1-21	Renames parameter keys to loralib-compatible format

CLI Signatures

split_lora.py

python utils/split_lora.py <input_checkpoint> <output_lora_path>

convert.py

python utils/convert.py <input_path> <output_path>

Input / Output

Script	Input	Output
`split_lora.py`	Full HF Trainer checkpoint (`pytorch_model.bin`)	Filtered checkpoint containing only LoRA matrices + classifier head
`convert.py`	Checkpoint with modified fork naming	Checkpoint with loralib-compatible naming
Combined pipeline	Full HF Trainer checkpoint	LoRA-only weights with loralib-compatible naming, ready for `--lora_path`

split_lora.py Implementation

The script loads the full checkpoint and filters parameters using two criteria:

import os
import sys
import torch
from collections import OrderedDict

if __name__ == "__main__":
    input_path = sys.argv[1]
    output_path = sys.argv[2]
    model = torch.load(input_path)

    lora = OrderedDict()
    for name, value in model.items():
        if 'lora' in name or (not name.startswith('deberta')
                               and not name.startswith('roberta')):
            new_name = name.replace(
                'self.query_lora_a.weight', 'self.query_proj.lora_A'
            ).replace(
                'self.query_lora_b.weight', 'self.query_proj.lora_B'
            ).replace(
                'self.value_lora_a.weight', 'self.value_proj.lora_A'
            ).replace(
                'self.value_lora_b.weight', 'self.value_proj.lora_B'
            )
            lora[new_name] = value
    print("Save:")
    print(lora.keys())
    folder = os.path.dirname(output_path)
    if not os.path.exists(folder):
        os.mkdir(folder)
    torch.save(lora, output_path)

Filtering Logic

The filter keeps a parameter if either of these conditions is true:

'lora' in name -- The parameter is a LoRA matrix (e.g., roberta.encoder.layer.0.attention.self.query.lora_A)
not name.startswith('deberta') and not name.startswith('roberta') -- The parameter is not part of the pretrained backbone (e.g., the classifier.dense.weight or classifier.out_proj.weight classification head)

This logic mirrors the parameter freezing in run_glue.py (lines 409-418), which freezes backbone parameters except those containing 'lora' and leaves non-backbone parameters trainable.

Simultaneous Renaming

Note that split_lora.py also performs the naming conversion (the same operation as convert.py) in the same pass. This means split_lora.py can serve as a single-step extraction + conversion tool.

convert.py Implementation

The convert.py script performs only the naming conversion, without any filtering:

import os
import sys
import torch
from collections import OrderedDict

if __name__ == "__main__":
    input_path = sys.argv[1]
    output_path = sys.argv[2]
    model = torch.load(input_path)

    lora = OrderedDict()
    for name, value in model.items():
        new_name = name.replace(
            'self.query_lora_a.weight', 'self.query_proj.lora_A'
        ).replace(
            'self.query_lora_b.weight', 'self.query_proj.lora_B'
        ).replace(
            'self.value_lora_a.weight', 'self.value_proj.lora_A'
        ).replace(
            'self.value_lora_b.weight', 'self.value_proj.lora_B'
        )
        lora[new_name] = value
    print("Save:")
    print(lora.keys())
    folder = os.path.dirname(output_path)
    if not os.path.exists(folder):
        os.mkdir(folder)
    torch.save(lora, output_path)

Key Renaming Map

Original Key Pattern	Converted Key Pattern
`self.query_lora_a.weight`	`self.query_proj.lora_A`
`self.query_lora_b.weight`	`self.query_proj.lora_B`
`self.value_lora_a.weight`	`self.value_proj.lora_A`
`self.value_lora_b.weight`	`self.value_proj.lora_B`

This conversion is necessary because the modified Transformers fork stores LoRA matrices as separate sub-module weights (with .weight suffix), while the standard loralib convention stores them as direct parameter attributes on the parent projection module.

Typical Usage

Single-Step (using split_lora.py alone)

# Extract LoRA weights and convert naming in one step
python utils/split_lora.py \
    ./output/roberta_base_mnli/model/pytorch_model.bin \
    ./output/roberta_base_mnli/model/lora_weights.pt

Two-Step (split then convert separately)

# Step 1: Extract LoRA weights (if split_lora.py did not rename)
python utils/split_lora.py \
    ./output/checkpoint/pytorch_model.bin \
    ./output/checkpoint/lora_raw.pt

# Step 2: Convert naming convention
python utils/convert.py \
    ./output/checkpoint/lora_raw.pt \
    ./output/checkpoint/lora_weights.pt

Using the Extracted Weights for Evaluation

python -m torch.distributed.launch --nproc_per_node=8 \
    examples/text-classification/run_glue.py \
    --model_name_or_path roberta-base \
    --task_name mnli --do_eval \
    --apply_lora --lora_r 8 --lora_alpha 16 \
    --lora_path ./output/roberta_base_mnli/model/lora_weights.pt

Output Directory Handling

Both scripts create the output directory if it does not exist:

folder = os.path.dirname(output_path)
if not os.path.exists(folder):
    os.mkdir(folder)

Note that os.mkdir only creates a single directory level. If the output path requires nested directories, they must be created manually beforehand.

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment