Principle:AUTOMATIC1111 Stable diffusion webui Network weight restoration

Knowledge Sources	LoRA: Low-Rank Adaptation of Large Language Models stable-diffusion-webui
Domains	Stable Diffusion, LoRA, Weight Management, Non-Destructive Editing, PyTorch
Last Updated	2026-02-08 00:00 GMT

Overview

Network weight restoration is the backup/restore pattern that ensures model weights can be cleanly returned to their original pretrained state after network adapter modifications, enabling non-destructive weight editing across multiple generation runs.

Description

When LoRA or LyCORIS networks modify a model's weight tensors, the original pretrained weights must be preserved so they can be restored before applying a different set of networks or when returning to an unmodified model. This is critical because:

Additive modification is not reversible: If weight W is modified to W + delta_A, applying a second network's delta_B requires starting from W (not W + delta_A), otherwise the result would be W + delta_A + delta_B instead of the intended W + delta_B.

Multiple networks must compose from baseline: When switching between network configurations (e.g., changing a prompt's LoRA tags), all previous modifications must be undone before the new set is applied.

Model integrity: Users may switch between LoRA-modified and unmodified generation, requiring the ability to return to the exact pretrained weights.

The backup/restore pattern works as follows:

Backup on first modification: The first time a layer's weights are about to be modified, the original weight tensor is copied to CPU memory and stored as a network_weights_backup attribute on the module. Similarly, the original bias (if present) is stored as network_bias_backup. This copy is made using weight.to(devices.cpu, copy=True) to avoid GPU memory overhead for the backup.

Restore before reapplication: Before applying a new set of network deltas, the system copies the backup weights back into the module's weight tensor using in-place copy_(). This returns the weight to its exact pretrained state, from which the new deltas can be cleanly applied.

Special handling for MultiheadAttention: PyTorch's MultiheadAttention stores Q, K, V weights in a combined in_proj_weight tensor and has a separate out_proj submodule. The backup stores a tuple of (in_proj_weight, out_proj.weight) for weights, and uses the out_proj.bias for bias backup.

Usage

Weight restoration is used transparently by the weight patching system. It is invoked:

• Before applying a new set of networks (when network_current_names changes)
• When the "functional" LoRA mode computes deltas on every forward pass
• When resetting cached weights during model state dict loading

The backup is created lazily (only when the first network modification occurs) and persists for the lifetime of the model in memory. Users do not interact with this system directly.

Theoretical Basis

Non-Destructive Modification Pattern

The pattern can be expressed formally:

STATE:
    W_original: stored in network_weights_backup (CPU)
    W_current:  stored in module.weight (GPU)
    applied_set: stored in network_current_names

BACKUP(module):
    if network_weights_backup is None:
        network_weights_backup = copy(module.weight) -> CPU
        network_bias_backup = copy(module.bias) -> CPU  (if bias exists)

RESTORE(module):
    if network_weights_backup is not None:
        module.weight.copy_(network_weights_backup)
        module.bias.copy_(network_bias_backup)  (if applicable)

APPLY(module, networks):
    wanted = identify(networks)
    if wanted != applied_set:
        RESTORE(module)
        for net in networks:
            delta = net.calc_updown(module.weight)
            module.weight += delta
        applied_set = wanted

Memory Efficiency

Backups are stored on CPU to minimize GPU memory usage. The store_weights_backup function explicitly creates a CPU copy:

store_weights_backup(weight):
    if weight is None:
        return None
    return weight.to(devices.cpu, copy=True)

Restoration uses in-place copy_() to avoid allocating new GPU tensors:

restore_weights_backup(obj, field, weight_backup):
    if weight_backup is None:
        setattr(obj, field, None)
        return
    getattr(obj, field).copy_(weight_backup)

This means the weight tensor on GPU is overwritten directly from the CPU backup, which is a single device-to-device copy operation.

Invariants

The system maintains the following invariants:

1. Backup immutability: Once created, network_weights_backup is never modified (only read during restore).
2. Backup-before-modify: A backup is always created before the first modification, enforced by the weight application code checking weights_backup is None before applying.
3. Restore-before-reapply: When the wanted network set differs from the current set, restoration always occurs before new deltas are applied.
4. Clean state on empty set: When no networks are loaded (empty loaded_networks), the weights are restored to their backed-up state.

Network Metadata in Generation Info

After networks are applied, their identifiers (names and short hashes) are embedded in the generation info text via p.extra_generation_params["Lora hashes"]. This enables reproducibility: the generation info contains the exact LoRA configuration used, allowing the same result to be reproduced by parsing the info text and re-activating the same networks.

Related Pages

Implemented By

• Implementation:AUTOMATIC1111_Stable_diffusion_webui_Network_restore_weights