Implementation:Online ml River Optim AdaBound
| Knowledge Sources | |
|---|---|
| Domains | Online_Learning, Optimization |
| Last Updated | 2026-02-08 16:00 GMT |
Overview
AdaBound is an adaptive gradient optimizer that dynamically bounds the learning rate, providing a smooth transition from adaptive methods to SGD.
Description
AdaBound combines the benefits of adaptive learning rate methods (like Adam) with the strong generalization properties of SGD. It achieves this by employing dynamic bounds on learning rates, which gradually transition from element-wise adaptive learning rates to a fixed learning rate schedule. The algorithm maintains first and second moment estimates (m and v) of gradients like Adam, but clips the effective learning rate between lower and upper bounds that converge over time. This prevents the learning rate from becoming too large or too small, addressing convergence issues sometimes seen with Adam while maintaining fast initial training.
Usage
Import from river.optim and use as an optimizer in any River model that accepts optimizers, such as linear models or neural networks. Best suited for problems where Adam may have convergence issues but you still want adaptive learning rates.
Code Reference
Source Location
- Repository: Online_ml_River
- File: river/optim/ada_bound.py
Signature
class AdaBound(optim.base.Optimizer):
def __init__(self, lr=1e-3, beta_1=0.9, beta_2=0.999, eps=1e-8, gamma=1e-3, final_lr=0.1):
...
Import
from river import optim
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| lr | float | No (default=1e-3) | Initial learning rate |
| beta_1 | float | No (default=0.9) | Exponential decay rate for first moment estimates |
| beta_2 | float | No (default=0.999) | Exponential decay rate for second moment estimates |
| eps | float | No (default=1e-8) | Small constant for numerical stability |
| gamma | float | No (default=1e-3) | Convergence speed of bounds |
| final_lr | float | No (default=0.1) | Final (SGD) learning rate |
Outputs
| Name | Type | Description |
|---|---|---|
| optimizer | AdaBound | Configured optimizer instance ready for model training |
Usage Examples
from river import datasets
from river import evaluate
from river import linear_model
from river import metrics
from river import optim
from river import preprocessing
# Create AdaBound optimizer
optimizer = optim.AdaBound()
# Use with a linear model
dataset = datasets.Phishing()
model = (
preprocessing.StandardScaler() |
linear_model.LogisticRegression(optimizer)
)
metric = metrics.F1()
# Evaluate
score = evaluate.progressive_val_score(dataset, model, metric)
print(score) # F1: 88.06%
# Custom parameters
optimizer = optim.AdaBound(
lr=0.01,
beta_1=0.95,
beta_2=0.999,
final_lr=0.05,
gamma=1e-2
)
model = linear_model.LogisticRegression(optimizer)