Principle:Haotian liu LLaVA Weight Delta Application
| Knowledge Sources | |
|---|---|
| Domains | Model_Distribution, Weight_Management |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
Technique that reconstructs a full model by adding stored weight differences (deltas) to a base model, enabling compact and license-compliant distribution of fine-tuned model weights.
Description
Weight delta application is the reconstruction half of a delta-based model distribution scheme. When a fine-tuned model cannot be redistributed directly (due to licensing restrictions on the base model), only the difference between the fine-tuned and base model weights is distributed. The recipient, who independently possesses the base model, reconstructs the full model by adding the delta back to the base weights. This principle is mathematically defined as: given base weights W_base and delta D = W_finetuned - W_base, the reconstruction is W_finetuned = W_base + D.
Special handling is required for parameters with dimension mismatches (e.g., vocabulary expansion adds new rows to embedding matrices) and for parameters that exist only in the fine-tuned model (e.g., newly added projection layers).
Usage
Use this principle when you need to distribute or obtain fine-tuned model weights where the base model has redistribution restrictions. This was the standard mechanism for early LLaVA releases built on LLaMA, and remains relevant for any scenario where full model redistribution is impractical or legally constrained.
Theoretical Basis
The core operation is element-wise addition of weight tensors:
Where was computed during the delta creation step.
Pseudo-code Logic:
# Abstract algorithm (NOT real implementation)
for name, delta_param in delta_model.parameters():
if name in base_model:
if shapes_match(delta_param, base_param):
delta_param += base_param
else:
# Handle vocabulary expansion: add overlapping portion
delta_param[:base_rows, :base_cols] += base_param
else:
# New parameters (e.g., mm_projector) — keep as-is
pass