Implementation:AUTOMATIC1111 Stable diffusion webui LearnRateScheduler

Knowledge Sources	stable-diffusion-webui
Domains	Learning Rate, Training Optimization, Textual Inversion, Hyperparameter Scheduling
Last Updated	2026-02-08 00:00 GMT

Overview

Concrete tool for parsing piecewise learning rate schedule strings and applying step-based rate transitions to a PyTorch optimizer during textual inversion training, provided by the AUTOMATIC1111 stable-diffusion-webui repository.

Description

LearnRateScheduler parses a schedule string such as "0.005:100, 0.0001:1000, 1e-5:10000" into a sequence of (rate, end_step) pairs and manages transitions between phases during training. It works together with the internal LearnScheduleIterator class which handles the actual parsing and iteration.

Key behaviors:

• Schedule parsing: The schedule string is split by commas, and each segment is split by colons. A segment with two parts (e.g., "0.005:100") defines a rate until a specific step. A segment with one part (e.g., "0.005") defines a constant rate for the remainder of training.
• Step filtering: When cur_step > 0 (resuming training), phases that end before or at cur_step are skipped, except when step is -1 (meaning "until max_steps").
• Step clamping: Phase end steps are clamped to max_steps, ensuring training stops at the right time even if the schedule extends further.
• Optimizer update: The apply() method checks whether a phase transition should occur and updates all optimizer parameter groups' lr values accordingly.
• Termination signaling: The finished flag is set to True when all schedule phases have been exhausted.

This class is used both for the main learning rate and for gradient clipping values (with verbose=False).

Usage

Use this scheduler when:

• Training textual inversion embeddings with step-based learning rate decay
• You need a simple piecewise schedule without external scheduler dependencies
• Resuming training from a checkpoint and needing to skip completed schedule phases
• Applying the same schedule format to gradient clipping values

Code Reference

Source Location

• Repository: stable-diffusion-webui
• File: modules/textual_inversion/learn_schedule.py
• Lines: L50-81 (LearnRateScheduler), L4-47 (LearnScheduleIterator)

Signature

class LearnRateScheduler:
    def __init__(self, learn_rate, max_steps, cur_step=0, verbose=True):
        ...

    def step(self, step_number):
        """Check if a phase transition should occur. Returns True if the rate changed."""
        ...

    def apply(self, optimizer, step_number):
        """Call step() and update all optimizer param groups if the rate changed."""
        ...

class LearnScheduleIterator:
    def __init__(self, learn_rate, max_steps, cur_step=0):
        """
        Parses learn_rate string like "0.005:100, 0.0001:1000, 1e-5:10000"
        into an iterable of (rate, end_step) tuples.
        """
        ...

Import

from modules.textual_inversion.learn_schedule import LearnRateScheduler

I/O Contract

Inputs

Name	Type	Required	Description
learn_rate	str	Yes	Comma-separated schedule string, e.g. "0.005:100, 0.0001:1000, 1e-5:10000". A single number like "0.005" means constant rate
max_steps	int	Yes	Maximum number of training steps; schedule phases are clamped to this value
cur_step	int	No	Current step number for resumption; phases ending at or before this step are skipped; defaults to 0
verbose	bool	No	If True, prints learning rate transitions to console via tqdm.tqdm.write; defaults to True

Outputs

Name	Type	Description
learn_rate (attribute)	float	The current active learning rate value
end_step (attribute)	int	The step number at which the current phase ends
finished (attribute)	bool	True when all schedule phases have been exhausted

Usage Examples

Basic Usage

from modules.textual_inversion.learn_schedule import LearnRateScheduler
import torch

# Create a piecewise schedule: 0.005 until step 100, then 0.0001 until step 1000
scheduler = LearnRateScheduler("0.005:100, 0.0001:1000", max_steps=1000)

# Create an optimizer
optimizer = torch.optim.AdamW([some_tensor], lr=scheduler.learn_rate)

# In the training loop, apply the schedule at each step
for step in range(1000):
    scheduler.apply(optimizer, step)
    if scheduler.finished:
        break
    # ... training step ...

Resuming from a Checkpoint

# Resume training from step 500 -- the first phase (0.005:100) is skipped
scheduler = LearnRateScheduler(
    "0.005:100, 0.0001:1000, 1e-5:5000",
    max_steps=5000,
    cur_step=500
)
# scheduler.learn_rate is now 0.0001 (the active phase at step 500)

Constant Learning Rate

# A single number without step boundary means constant rate
scheduler = LearnRateScheduler("0.001", max_steps=10000)
# scheduler.learn_rate == 0.001 for all steps

Related Pages

Implements Principle

• Principle:AUTOMATIC1111_Stable_diffusion_webui_Learning_rate_scheduling