Principle:Haotian liu LLaVA Weight Delta Computation
| Knowledge Sources | |
|---|---|
| Domains | Model_Distribution, Weight_Management |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
Technique that computes the difference between a fine-tuned model and its base model weights, producing a compact delta for efficient and license-compliant model distribution.
Description
Weight delta computation is the creation half of a delta-based model distribution scheme. When a fine-tuned model cannot be redistributed in full (due to licensing restrictions on the base model), this technique computes and stores only the difference between the fine-tuned and base model weights. The resulting delta is significantly smaller in semantic information (though the same size in bytes) and can be legally distributed because it does not contain the original base weights.
Special handling is required for parameters with dimension mismatches (e.g., vocabulary expansion) and for parameters that exist only in the fine-tuned model (e.g., newly added projection layers like mm_projector), which are stored as-is.
Usage
Use this principle when you have trained a model on top of a base model with redistribution restrictions and need to share your fine-tuned weights. This is the companion to weight delta application: you compute the delta once, and users apply it to reconstruct the full model.
Theoretical Basis
The core operation is element-wise subtraction of weight tensors:
The resulting delta can then be distributed and later used to reconstruct the full model via .
Pseudo-code Logic:
# Abstract algorithm (NOT real implementation)
for name, target_param in finetuned_model.parameters():
if name in base_model:
if shapes_match(target_param, base_param):
target_param -= base_param
else:
# Handle vocabulary expansion: subtract overlapping portion
target_param[:base_rows, :base_cols] -= base_param
else:
# New parameters (e.g., mm_projector) — keep as-is
pass
# Save delta to disk or push to hub