Implementation:AUTOMATIC1111 Stable diffusion webui Network apply weights
| Knowledge Sources | |
|---|---|
| Domains | Stable Diffusion, LoRA, Monkey-Patching, Weight Injection, PyTorch |
| Last Updated | 2026-02-08 00:00 GMT |
Overview
Concrete tool for applying loaded network weight deltas to model layers via monkey-patched forward methods, provided by the AUTOMATIC1111 stable-diffusion-webui repository.
Description
This implementation consists of two cooperating components:
network_apply_weights() is the core function that performs lazy weight modification on a single torch.nn.Module. When invoked, it compares the current set of loaded networks (identified by name, TE multiplier, UNet multiplier, and dynamic dimension) against the set already applied to this layer. If they differ, it restores original weights from backup, then iterates over all loaded networks, retrieves the matching NetworkModule for this layer, calls calc_updown(weight) to compute the low-rank delta, and adds it to the layer's weight tensor. For MultiheadAttention layers, it handles the combined in_proj_weight (Q/K/V) and separate out_proj individually. It also supports inpainting models by zero-padding updown tensors from 4 to 9 channels and handles fp16 weight caching.
LoraPatches is the monkey-patching class that intercepts forward calls on all supported PyTorch layer types. It replaces the forward() and _load_from_state_dict() methods of Linear, Conv2d, GroupNorm, LayerNorm, and MultiheadAttention with custom versions that call network_apply_weights() before delegating to the original implementation. It also provides an undo() method to restore all original methods.
Usage
LoraPatches is instantiated once during the Lora extension's initialization and remains active for the application lifetime. network_apply_weights() is called implicitly on every forward pass of every supported layer type in the model. External code should not call these functions directly.
Code Reference
Source Location
- Repository: stable-diffusion-webui
- File (network_apply_weights):
extensions-builtin/Lora/networks.py - Lines: 411-542
- File (LoraPatches):
extensions-builtin/Lora/lora_patches.py - Lines: 7-30
Signature
# extensions-builtin/Lora/networks.py
def network_apply_weights(self: Union[torch.nn.Conv2d, torch.nn.Linear,
torch.nn.GroupNorm, torch.nn.LayerNorm,
torch.nn.MultiheadAttention]):
"""
Applies the currently selected set of networks to the weights of torch layer self.
If weights already have this particular set of networks applied, does nothing.
If not, restores original weights from backup and alters weights according to networks.
"""
...
# extensions-builtin/Lora/lora_patches.py
class LoraPatches:
def __init__(self):
self.Linear_forward = patches.patch(__name__, torch.nn.Linear, 'forward',
networks.network_Linear_forward)
self.Linear_load_state_dict = patches.patch(__name__, torch.nn.Linear,
'_load_from_state_dict',
networks.network_Linear_load_state_dict)
self.Conv2d_forward = patches.patch(__name__, torch.nn.Conv2d, 'forward',
networks.network_Conv2d_forward)
self.Conv2d_load_state_dict = patches.patch(__name__, torch.nn.Conv2d,
'_load_from_state_dict',
networks.network_Conv2d_load_state_dict)
self.GroupNorm_forward = patches.patch(__name__, torch.nn.GroupNorm, 'forward',
networks.network_GroupNorm_forward)
self.GroupNorm_load_state_dict = patches.patch(__name__, torch.nn.GroupNorm,
'_load_from_state_dict',
networks.network_GroupNorm_load_state_dict)
self.LayerNorm_forward = patches.patch(__name__, torch.nn.LayerNorm, 'forward',
networks.network_LayerNorm_forward)
self.LayerNorm_load_state_dict = patches.patch(__name__, torch.nn.LayerNorm,
'_load_from_state_dict',
networks.network_LayerNorm_load_state_dict)
self.MultiheadAttention_forward = patches.patch(__name__, torch.nn.MultiheadAttention,
'forward',
networks.network_MultiheadAttention_forward)
self.MultiheadAttention_load_state_dict = patches.patch(__name__, torch.nn.MultiheadAttention,
'_load_from_state_dict',
networks.network_MultiheadAttention_load_state_dict)
def undo(self):
self.Linear_forward = patches.undo(__name__, torch.nn.Linear, 'forward')
self.Linear_load_state_dict = patches.undo(__name__, torch.nn.Linear, '_load_from_state_dict')
self.Conv2d_forward = patches.undo(__name__, torch.nn.Conv2d, 'forward')
self.Conv2d_load_state_dict = patches.undo(__name__, torch.nn.Conv2d, '_load_from_state_dict')
self.GroupNorm_forward = patches.undo(__name__, torch.nn.GroupNorm, 'forward')
self.GroupNorm_load_state_dict = patches.undo(__name__, torch.nn.GroupNorm, '_load_from_state_dict')
self.LayerNorm_forward = patches.undo(__name__, torch.nn.LayerNorm, 'forward')
self.LayerNorm_load_state_dict = patches.undo(__name__, torch.nn.LayerNorm, '_load_from_state_dict')
self.MultiheadAttention_forward = patches.undo(__name__, torch.nn.MultiheadAttention, 'forward')
self.MultiheadAttention_load_state_dict = patches.undo(__name__, torch.nn.MultiheadAttention, '_load_from_state_dict')
Import
import networks
from lora_patches import LoraPatches
# network_apply_weights is called indirectly through patched forward methods
# LoraPatches is instantiated once during extension setup
originals = LoraPatches()
I/O Contract
Inputs
network_apply_weights:
| Name | Type | Required | Description |
|---|---|---|---|
| self | Union[torch.nn.Conv2d, torch.nn.Linear, torch.nn.GroupNorm, torch.nn.LayerNorm, torch.nn.MultiheadAttention] | Yes | The PyTorch module whose weights should be patched with the current loaded networks |
LoraPatches.__init__:
| Name | Type | Required | Description |
|---|---|---|---|
| (none) | -- | -- | No parameters; patches are applied to global PyTorch module classes upon construction |
Outputs
network_apply_weights:
| Name | Type | Description |
|---|---|---|
| return | None | Side effect: modifies self.weight (and optionally self.bias) in-place by adding computed deltas from all loaded networks; creates backup attributes network_weights_backup and network_bias_backup on first call; updates network_current_names to track applied state
|
LoraPatches:
| Name | Type | Description |
|---|---|---|
| (attributes) | callable | Each attribute stores the original method reference returned by patches.patch(), used for delegation and undo
|
Usage Examples
Basic Usage
# The patching is transparent to normal model usage.
# After LoraPatches is initialized and networks are loaded:
import networks
import torch
# Assume networks are loaded via the activation pipeline:
networks.load_networks(["my_lora"], te_multipliers=[0.8], unet_multipliers=[0.8])
# When the model runs its forward pass, each layer's patched forward
# automatically calls network_apply_weights before the original computation:
#
# model_output = sd_model(latent_input, timestep, context)
#
# Internally, for each Linear/Conv2d/etc layer:
# 1. network_apply_weights(layer) checks if weights need updating
# 2. If network set changed: restore backup, apply all network deltas
# 3. original_forward(layer, input) runs with modified weights
Patched Forward Flow
# Example of what happens inside a patched Linear forward:
def network_Linear_forward(self, input):
if shared.opts.lora_functional:
return network_forward(self, input, originals.Linear_forward)
# Default path: modify weights, then run original forward
network_apply_weights(self)
return originals.Linear_forward(self, input)
Weight Delta Computation
# For each network module matching this layer:
with torch.no_grad():
weight = self.weight
updown, ex_bias = module.calc_updown(weight)
# For inpainting models with 9-channel input:
if len(weight.shape) == 4 and weight.shape[1] == 9:
updown = torch.nn.functional.pad(updown, (0, 0, 0, 0, 0, 5))
# Apply the delta
self.weight.copy_((weight.to(dtype=updown.dtype) + updown).to(dtype=self.weight.dtype))
# Apply bias delta if present
if ex_bias is not None and hasattr(self, 'bias'):
if self.bias is None:
self.bias = torch.nn.Parameter(ex_bias).to(self.weight.dtype)
else:
self.bias.copy_((bias + ex_bias).to(dtype=self.bias.dtype))