Principle:Haotian liu LLaVA LoRA Weight Merging
Overview
Technique for permanently combining LoRA adapter weights with the base model to create a standalone model without adapter overhead.
Description
After LoRA training, the model exists as a base model plus separate adapter weights (stored in adapter_model.bin and non_lora_trainables.bin). LoRA weight merging loads the base model, applies the LoRA adapters, calls merge_and_unload() to permanently fold the adapter weights back into the original weight matrices, and saves the result as a standard HuggingFace checkpoint.
The merging process follows these steps:
- Load the base LLaVA model (LlavaLlamaForCausalLM.from_pretrained())
- Load non_lora_trainables.bin and apply to the base model via load_state_dict(strict=False)
- Load LoRA adapter weights via PeftModel.from_pretrained()
- Call merge_and_unload() to permanently fold adapters into base weights
- Save the merged model and tokenizer as a standard HuggingFace checkpoint
This eliminates the runtime overhead of LoRA inference and removes the dependency on the peft library for deployment.
Usage
Use this after LoRA training to create a deployable model. The merged model can be:
- Used directly with standard HuggingFace inference pipelines
- Served via LLaVA's model_worker without specifying --model-base
- Evaluated on benchmarks without requiring LoRA-aware loading
- Shared or distributed as a self-contained model checkpoint
Merging is not required for inference -- LLaVA can load unmerged LoRA models by specifying --model-base. However, merging simplifies deployment and eliminates the need to distribute the base model separately.
Theoretical Basis
merge_and_unload() computes the following for each adapted layer:
- W' = W + B * A * (alpha / r)
where:
- W is the original frozen weight matrix
- B and A are the LoRA adapter matrices
- alpha / r is the LoRA scaling factor
After merging, the LoRA-wrapped module is replaced with a standard nn.Linear module containing W' . The non_lora_trainables.bin file (containing mm_projector weights) is loaded separately via load_state_dict(strict=False) before the LoRA adapter is applied and merged.
The merged model is mathematically equivalent to the LoRA model at inference time. The only difference is that the merged model stores the full combined weights rather than the factored representation, trading storage efficiency for inference simplicity.
Knowledge Sources
- Paper -- LoRA: Low-Rank Adaptation of Large Language Models -- https://arxiv.org/abs/2106.09685
Domains
- Model_Deployment
- Parameter_Efficient_Fine_Tuning
Metadata
| Field | Value |
|---|---|
| last_updated | 2026-02-13 14:00 GMT |
| source_repo | Haotian_liu_LLaVA |
| commit | 799f5f207c89 |
| type | Principle |