Implementation:Haotian liu LLaVA Make Delta
| Knowledge Sources | |
|---|---|
| Domains | Model_Distribution, Weight_Management, Vision_Language |
| Last Updated | 2026-02-14 00:00 GMT |
Overview
Concrete tool for computing a weight delta between a fine-tuned LLaVA model and its base LLaMA model, producing a compact diff for distribution.
Description
The make_delta function loads a base LLaMA model and a target (fine-tuned) LLaVA model, then subtracts the base weights from the target weights parameter-by-parameter. For parameters present in both models with matching shapes, it performs element-wise subtraction. For dimension mismatches in embed_tokens.weight and lm_head.weight (caused by vocabulary expansion), it subtracts only the overlapping portion. Parameters unique to LLaVA (mm_projector.weight/bias) are preserved unchanged. The resulting delta can be saved locally or pushed directly to HuggingFace Hub. This is the inverse operation of apply_delta.
Usage
Use this tool when you have trained a LLaVA model and need to distribute it as a compact delta relative to the base LLaMA model. This was essential for early LLaVA releases where LLaMA license restrictions prevented full model weight redistribution.
Code Reference
Source Location
- Repository: Haotian_liu_LLaVA
- File: llava/model/make_delta.py
- Lines: 1-52
Signature
def make_delta(base_model_path: str, target_model_path: str, delta_path: str, hub_repo_id: str) -> None:
"""
Compute a weight delta between a fine-tuned LLaVA model and its base LLaMA model.
Args:
base_model_path: Path to the base LLaMA model weights.
target_model_path: Path to the fine-tuned LLaVA model weights.
delta_path: Path where the computed delta will be saved.
hub_repo_id: Optional HuggingFace Hub repo ID for push_to_hub.
"""
Import
from llava.model.make_delta import make_delta
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| base_model_path | str | Yes | Filesystem path to the base LLaMA model (e.g., llama-7b) |
| target_model_path | str | Yes | Filesystem path to the fine-tuned LLaVA model |
| delta_path | str | Yes | Filesystem path where the delta will be saved |
| hub_repo_id | str | No | HuggingFace Hub repository ID for automatic upload (e.g., liuhaotian/llava-7b-delta) |
Outputs
| Name | Type | Description |
|---|---|---|
| Saved delta | Files | Delta model weights saved to delta_path |
| Saved tokenizer | Files | Tokenizer files saved to delta_path |
| Hub upload | Optional | If hub_repo_id is provided, delta is pushed to HuggingFace Hub |
Usage Examples
CLI Usage
# Compute delta and save locally
python3 -m llava.model.make_delta \
--base-model-path ~/model_weights/llama-7b \
--target-model-path ~/model_weights/llava-7b \
--delta-path ~/model_weights/llava-7b-delta
# Compute delta and push to HuggingFace Hub
python3 -m llava.model.make_delta \
--base-model-path ~/model_weights/llama-7b \
--target-model-path ~/model_weights/llava-7b \
--delta-path ~/model_weights/llava-7b-delta \
--hub-repo-id liuhaotian/llava-7b-delta
Programmatic Usage
from llava.model.make_delta import make_delta
# Compute and save delta locally
make_delta(
base_model_path="/path/to/llama-7b",
target_model_path="/path/to/llava-7b",
delta_path="/path/to/output/llava-7b-delta",
hub_repo_id=None
)
# Compute delta and push to HuggingFace Hub
make_delta(
base_model_path="/path/to/llama-7b",
target_model_path="/path/to/llava-7b",
delta_path="/path/to/output/llava-7b-delta",
hub_repo_id="liuhaotian/llava-7b-delta"
)