Implementation:Haotian liu LLaVA Merge LoRA Weights

Overview

CLI utility for merging LoRA adapter weights with the base LLaVA model into a standalone checkpoint.

Type

API Doc

Description

merge_lora() loads the LoRA adapter using load_pretrained_model() (which internally handles non_lora_trainables.bin loading, PeftModel.from_pretrained(), and merge_and_unload()), then saves the merged model and tokenizer to the specified output directory.

The LoRA loading path inside load_pretrained_model() (in llava/model/builder.py:L52-86) performs these steps:

1. Detects LoRA model by checking for "lora" in the model name
2. Loads the base model via LlavaLlamaForCausalLM.from_pretrained(model_base)
3. Loads non_lora_trainables.bin from the adapter directory (or HuggingFace Hub)
4. Cleans key prefixes (removes "base_model." and "model." prefixes as needed)
5. Applies non-LoRA weights via model.load_state_dict(non_lora_trainables, strict=False)
6. Loads LoRA adapter via PeftModel.from_pretrained(model, model_path)
7. Calls model.merge_and_unload() to permanently fold adapters into weights

Source

• scripts/merge_lora_weights.py:L6-11 (merge_lora function)
• llava/model/builder.py:L52-86 (LoRA loading path in load_pretrained_model)

Signature

def merge_lora(args) -> None:
    """Merge LoRA adapter weights with base model.

    Args:
        args.model_path (str): Path to LoRA adapter checkpoint directory
        args.model_base (str): Path or HF ID of the base LLaVA model
        args.save_model_path (str): Output directory for merged model
    """
    model_name = get_model_name_from_path(args.model_path)
    tokenizer, model, image_processor, context_len = load_pretrained_model(
        args.model_path, args.model_base, model_name, device_map='cpu'
    )
    model.save_pretrained(args.save_model_path)
    tokenizer.save_pretrained(args.save_model_path)

LoRA Loading Path in builder.py (L52-86)

if 'lora' in model_name.lower() and model_base is not None:
    from llava.model.language_model.llava_llama import LlavaConfig
    lora_cfg_pretrained = LlavaConfig.from_pretrained(model_path)
    tokenizer = AutoTokenizer.from_pretrained(model_base, use_fast=False)
    model = LlavaLlamaForCausalLM.from_pretrained(
        model_base, low_cpu_mem_usage=True, config=lora_cfg_pretrained, **kwargs
    )

    # Load non-LoRA trainables (mm_projector weights)
    if os.path.exists(os.path.join(model_path, 'non_lora_trainables.bin')):
        non_lora_trainables = torch.load(
            os.path.join(model_path, 'non_lora_trainables.bin'), map_location='cpu'
        )
    else:
        from huggingface_hub import hf_hub_download
        non_lora_trainables = load_from_hf(model_path, 'non_lora_trainables.bin')

    non_lora_trainables = {
        (k[11:] if k.startswith('base_model.') else k): v
        for k, v in non_lora_trainables.items()
    }
    model.load_state_dict(non_lora_trainables, strict=False)

    from peft import PeftModel
    model = PeftModel.from_pretrained(model, model_path)
    model = model.merge_and_unload()

Import

# CLI script -- typically invoked directly
from scripts.merge_lora_weights import merge_lora

CLI Usage

python scripts/merge_lora_weights.py \
    --model-path /path/to/lora-adapter-checkpoint \
    --model-base liuhaotian/llava-v1.5-13b \
    --save-model-path /path/to/merged-model-output

Inputs

Argument	Type	Required	Description
--model-path	str	Yes	Path to LoRA adapter checkpoint directory (contains adapter_config.json, adapter_model.bin, non_lora_trainables.bin)
--model-base	str	Yes	Path or HuggingFace model ID of the base LLaVA model used during LoRA training
--save-model-path	str	Yes	Output directory for the merged model

Outputs

Merged HuggingFace-compatible model directory containing:

• config.json -- Model configuration
• pytorch_model.bin (or model.safetensors) -- Full merged model weights
• tokenizer.json, tokenizer_config.json -- Tokenizer files
• special_tokens_map.json -- Special token mappings
• generation_config.json -- Generation configuration

The output directory can be loaded directly with LlavaLlamaForCausalLM.from_pretrained() without specifying any LoRA or base model arguments.

Usage Example

Complete Merge Workflow

# Step 1: After LoRA training, merge the adapter with the base model
python scripts/merge_lora_weights.py \
    --model-path ./checkpoints/llava-v1.5-13b-task-lora \
    --model-base liuhaotian/llava-v1.5-13b \
    --save-model-path ./checkpoints/llava-v1.5-13b-task-merged

# Step 2: Use the merged model directly (no --model-base needed)
python -m llava.eval.run_llava \
    --model-path ./checkpoints/llava-v1.5-13b-task-merged \
    --image-file test.jpg \
    --query "Describe this image."

Metadata

Field	Value
last_updated	2026-02-13 14:00 GMT
source_repo	Haotian_liu_LLaVA
commit	799f5f207c89
type	Implementation (API Doc)

Related Pages

• implements Principle:Haotian_liu_LLaVA_LoRA_Weight_Merging