Implementation:Online ml River Optim AMSGrad

Knowledge Sources	Online_ml_River
Domains	Online_Learning, Optimization
Last Updated	2026-02-08 16:00 GMT

Overview

AMSGrad is a variant of Adam that addresses convergence issues by maintaining the maximum of past second moment estimates.

Description

AMSGrad fixes a potential convergence problem in Adam by using the maximum of all past second moment estimates rather than the exponentially decaying average. This prevents the learning rate from increasing and ensures that the algorithm converges to better solutions. The key modification is maintaining v_hat, which tracks the maximum of the second moment estimates over time. By using max(v_hat, v) instead of just v in the denominator of the update rule, AMSGrad guarantees that the effective step size never increases, which can lead to better convergence properties. The algorithm optionally supports bias correction like Adam through the correct_bias parameter.

Usage

Import from river.optim and use as an optimizer in any River model. Prefer AMSGrad over Adam when convergence to optimal solutions is critical.

Code Reference

Source Location

• Repository: Online_ml_River
• File: river/optim/ams_grad.py

Signature

class AMSGrad(optim.base.Optimizer):
    def __init__(
        self,
        lr: int | float | optim.base.Scheduler = 0.1,
        beta_1=0.9,
        beta_2=0.999,
        eps=1e-8,
        correct_bias=True,
    ):
        ...

Import

from river import optim

I/O Contract

Inputs

Name	Type	Required	Description
lr	int, float, or Scheduler	No (default=0.1)	Learning rate or learning rate scheduler
beta_1	float	No (default=0.9)	Exponential decay rate for first moment estimates
beta_2	float	No (default=0.999)	Exponential decay rate for second moment estimates
eps	float	No (default=1e-8)	Small constant for numerical stability
correct_bias	bool	No (default=True)	Whether to apply bias correction

Outputs

Name	Type	Description
optimizer	AMSGrad	Configured optimizer instance ready for model training

Usage Examples

from river import datasets
from river import evaluate
from river import linear_model
from river import metrics
from river import optim
from river import preprocessing

# Create AMSGrad optimizer
optimizer = optim.AMSGrad()

# Use with a linear model
dataset = datasets.Phishing()
model = (
    preprocessing.StandardScaler() |
    linear_model.LogisticRegression(optimizer)
)
metric = metrics.F1()

# Evaluate
score = evaluate.progressive_val_score(dataset, model, metric)
print(score)  # F1: 86.60%

# Custom parameters
optimizer = optim.AMSGrad(
    lr=0.01,
    beta_1=0.95,
    beta_2=0.999,
    eps=1e-7,
    correct_bias=False
)

model = linear_model.LogisticRegression(optimizer)

# With learning rate scheduler
from river.optim import schedulers

scheduler = schedulers.InverseScaling(learning_rate=0.1, power=0.5)
optimizer = optim.AMSGrad(lr=scheduler)
model = linear_model.LogisticRegression(optimizer)

Related Pages

• Environment:Online_ml_River_Python_Runtime_Environment