Implementation:AUTOMATIC1111 Stable diffusion webui Network restore weights

Knowledge Sources	stable-diffusion-webui
Domains	Stable Diffusion, LoRA, Weight Management, Non-Destructive Editing, PyTorch
Last Updated	2026-02-08 00:00 GMT

Overview

Concrete tool for restoring model layer weights from CPU-stored backups after network adapter modifications, provided by the AUTOMATIC1111 stable-diffusion-webui repository.

Description

network_restore_weights_from_backup() is the function responsible for returning a model layer's weight and bias tensors to their original pretrained values. It reads from the network_weights_backup and network_bias_backup attributes previously stored on the module by the weight application system, and copies them back into the module's active weight/bias tensors using in-place operations.

The function handles two distinct cases:

Standard layers (Linear, Conv2d, GroupNorm, LayerNorm): The backup is a single tensor. Restoration copies it directly into self.weight and self.bias.

MultiheadAttention layers: The weight backup is a tuple of two tensors: (in_proj_weight_backup, out_proj_weight_backup). Restoration copies the first into self.in_proj_weight and the second into self.out_proj.weight. The bias backup corresponds to self.out_proj.bias.

Supporting helper functions store_weights_backup() and restore_weights_backup() handle the CPU copy and in-place restoration respectively.

Usage

This function is called by network_apply_weights() whenever the set of loaded networks changes (detected by comparing network_current_names with the wanted set). It is also called by the functional forward path (network_forward()) on every forward pass. External code should not call this function directly; it is part of the internal weight management lifecycle.

Code Reference

Source Location

• Repository: stable-diffusion-webui
• File: extensions-builtin/Lora/networks.py
• Lines: 376-408 (network_restore_weights_from_backup), 376-380 (store_weights_backup), 383-388 (restore_weights_backup)

Signature

def store_weights_backup(weight):
    if weight is None:
        return None
    return weight.to(devices.cpu, copy=True)


def restore_weights_backup(obj, field, weight):
    if weight is None:
        setattr(obj, field, None)
        return
    getattr(obj, field).copy_(weight)


def network_restore_weights_from_backup(
    self: Union[torch.nn.Conv2d, torch.nn.Linear, torch.nn.GroupNorm,
                torch.nn.LayerNorm, torch.nn.MultiheadAttention]
):
    weights_backup = getattr(self, "network_weights_backup", None)
    bias_backup = getattr(self, "network_bias_backup", None)

    if weights_backup is None and bias_backup is None:
        return

    if weights_backup is not None:
        if isinstance(self, torch.nn.MultiheadAttention):
            restore_weights_backup(self, 'in_proj_weight', weights_backup[0])
            restore_weights_backup(self.out_proj, 'weight', weights_backup[1])
        else:
            restore_weights_backup(self, 'weight', weights_backup)

    if isinstance(self, torch.nn.MultiheadAttention):
        restore_weights_backup(self.out_proj, 'bias', bias_backup)
    else:
        restore_weights_backup(self, 'bias', bias_backup)

Import

import networks
# Called internally; not typically imported directly
networks.network_restore_weights_from_backup(module)

I/O Contract

Inputs

network_restore_weights_from_backup:

Name	Type	Required	Description
self	Union[torch.nn.Conv2d, torch.nn.Linear, torch.nn.GroupNorm, torch.nn.LayerNorm, torch.nn.MultiheadAttention]	Yes	The PyTorch module whose weights should be restored from backup

store_weights_backup:

Name	Type	Required	Description
weight	torch.Tensor or None	Yes	The weight tensor to backup; None is handled gracefully

restore_weights_backup:

Name	Type	Required	Description
obj	torch.nn.Module	Yes	The module or submodule containing the target attribute
field	str	Yes	The attribute name to restore (e.g., 'weight', 'bias', 'in_proj_weight')
weight	torch.Tensor or None	Yes	The backup tensor to copy from; if None, the field is set to None

Outputs

network_restore_weights_from_backup:

Name	Type	Description
return	None	Side effect: restores self.weight and self.bias (or self.in_proj_weight, self.out_proj.weight, self.out_proj.bias for MHA) to their backed-up pretrained values using in-place copy_()

store_weights_backup:

Name	Type	Description
return	torch.Tensor or None	A CPU copy of the input weight tensor, or None if input was None

Usage Examples

Basic Usage

import networks
import torch

# Assume a Linear layer with network modifications applied:
layer = torch.nn.Linear(768, 768)
# ... weights have been modified by network_apply_weights ...

# The backup was automatically created on first modification:
# layer.network_weights_backup contains the original weight on CPU
# layer.network_bias_backup contains the original bias on CPU (or None)

# Restore to original pretrained weights:
networks.network_restore_weights_from_backup(layer)

# layer.weight now contains the original pretrained values
# layer.bias now contains the original pretrained values

MultiheadAttention Restoration

import networks
import torch

# MultiheadAttention stores combined Q/K/V in in_proj_weight
mha_layer = torch.nn.MultiheadAttention(embed_dim=768, num_heads=12)

# After network modifications, the backup is a tuple:
# mha_layer.network_weights_backup = (in_proj_weight_cpu, out_proj_weight_cpu)
# mha_layer.network_bias_backup = out_proj_bias_cpu

# Restore all three weight tensors:
networks.network_restore_weights_from_backup(mha_layer)

# mha_layer.in_proj_weight restored from weights_backup[0]
# mha_layer.out_proj.weight restored from weights_backup[1]
# mha_layer.out_proj.bias restored from bias_backup

Reset Cached Weight State

import networks

# network_reset_cached_weight clears the tracking state,
# forcing reapplication on the next forward pass:
def network_reset_cached_weight(self):
    self.network_current_names = ()
    self.network_weights_backup = None
    self.network_bias_backup = None

# This is called automatically when _load_from_state_dict is invoked
# (e.g., when switching the base model), ensuring fresh backups are
# created from the new model's weights.

Related Pages

Implements Principle

• Principle:AUTOMATIC1111_Stable_diffusion_webui_Network_weight_restoration