Implementation:Lm sys FastChat Apply Delta Weights

Knowledge Sources	Lm_sys_FastChat
Domains	Model Weights, LLM, Weight Management
Last Updated	2026-02-07 06:00 GMT

Overview

Applies delta weights on top of a base model to reconstruct a target model, supporting both standard and memory-efficient processing modes.

Description

The apply_delta module provides the mechanism to reconstruct fine-tuned models (such as Vicuna) from their base model (such as LLaMA) and a set of delta weights. This approach allows distributing only the weight differences rather than full model weights, which is critical for complying with base model licenses that prohibit direct redistribution.

The module offers two execution paths. The standard apply_delta function loads both the base model and the delta weights fully into memory using HuggingFace's AutoModelForCausalLM, iterates over all parameters, and adds the delta to the base in-place. This is straightforward but requires sufficient RAM to hold two complete models simultaneously.

For environments with limited memory, the apply_delta_low_cpu_mem function splits both the base and delta weight files into 4GB chunks using split_files, writes them to temporary directories, and then processes the chunks one at a time. This disk-backed approach reduces peak memory usage below 10GB at the cost of additional I/O overhead. The split_files helper also supports downloading models directly from HuggingFace Hub via snapshot_download if the model path does not exist locally.

Usage

Use this module when you need to reconstruct a fine-tuned model from its base and delta weights. This is the standard approach for obtaining Vicuna model weights. Choose the --low-cpu-mem flag when working on machines with limited RAM (under 32GB). The module can be run directly from the command line as python3 -m fastchat.model.apply_delta.

Code Reference

Source Location

Repository: Lm_sys_FastChat
File: fastchat/model/apply_delta.py
Lines: 1-165

Signature

def split_files(model_path: str, tmp_path: str, split_size: int) -> None:
    """Splits large weight files into smaller chunks for low-memory processing."""
    ...

def apply_delta_low_cpu_mem(base_model_path: str, target_model_path: str, delta_path: str) -> None:
    """Memory-efficient delta application using chunked file processing."""
    ...

def apply_delta(base_model_path: str, target_model_path: str, delta_path: str) -> None:
    """Standard delta application loading full models into memory."""
    ...

Import

from fastchat.model.apply_delta import apply_delta

I/O Contract

Inputs

Name	Type	Required	Description
base_model_path	str	Yes	Path to the base model weights directory or HuggingFace Hub model ID (e.g., llama-7b)
target_model_path	str	Yes	Output path where the reconstructed target model will be saved
delta_path	str	Yes	Path to the delta weights directory or HuggingFace Hub model ID (e.g., lmsys/vicuna-7b-delta-v1.1)

Outputs

Name	Type	Description
target_model_path (on disk)	Directory	Saved model directory containing the reconstructed model weights, tokenizer, and config files

Usage Examples

# Command-line usage (standard mode)
# python3 -m fastchat.model.apply_delta \
#     --base ~/model_weights/llama-7b \
#     --target ~/model_weights/vicuna-7b \
#     --delta lmsys/vicuna-7b-delta-v1.1

# Command-line usage (low memory mode)
# python3 -m fastchat.model.apply_delta \
#     --base ~/model_weights/llama-7b \
#     --target ~/model_weights/vicuna-7b \
#     --delta lmsys/vicuna-7b-delta-v1.1 \
#     --low-cpu-mem

# Programmatic usage
from fastchat.model.apply_delta import apply_delta, apply_delta_low_cpu_mem

# Standard application (requires ~32GB+ RAM for 13B models)
apply_delta(
    base_model_path="~/model_weights/llama-7b",
    target_model_path="~/model_weights/vicuna-7b",
    delta_path="lmsys/vicuna-7b-delta-v1.1"
)

# Low-memory application (keeps usage under 10GB)
apply_delta_low_cpu_mem(
    base_model_path="~/model_weights/llama-13b",
    target_model_path="~/model_weights/vicuna-13b",
    delta_path="lmsys/vicuna-13b-delta-v1.1"
)

Related Pages

Principle:Lm_sys_FastChat_Model_Weight_Delta_Distribution
Implements: Principle:Lm_sys_FastChat_Model_Weight_Delta_Distribution
Lm_sys_FastChat_Make_Delta_Weights - Creates delta weights (the inverse operation)
Lm_sys_FastChat_Condense_Rotary_Embedding - Another model modification technique in the same module directory
Lm_sys_FastChat_Huggingface_API_Inference - Uses the reconstructed models for inference

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment