Principle:Online ml River Online Neural Networks
| Knowledge Sources | |
|---|---|
| Domains | Online_Learning, Neural_Networks |
| Last Updated | 2026-02-08 18:00 GMT |
Overview
Online neural networks adapt the standard multi-layer perceptron (MLP) and related architectures to process data one example at a time, updating weights after each observation rather than accumulating mini-batches. This makes neural networks applicable to streaming environments where data arrives continuously and storing large datasets is infeasible.
The core mechanism is stochastic gradient descent (SGD) with backpropagation applied to a single sample at each step. While noisier than batch training, online SGD provides an unbiased gradient estimate and can track non-stationary distributions.
Theoretical Basis
Feedforward Architecture
An MLP with L hidden layers computes:
h_0 = x
h_l = activation(W_l * h_{l-1} + b_l), l = 1, ..., L
y_hat = W_{L+1} * h_L + b_{L+1}
Each weight matrix W_l and bias vector b_l is updated via gradient descent on the loss for the current sample.
Activation Functions
Activation functions introduce non-linearity, enabling the network to approximate complex functions. Common choices include:
- ReLU: max(0, z) -- computationally efficient, avoids vanishing gradient for positive inputs.
- Sigmoid: 1 / (1 + exp(-z)) -- maps to (0, 1), historically popular but prone to saturation.
- Tanh: (exp(z) - exp(-z)) / (exp(z) + exp(-z)) -- zero-centered variant of sigmoid.
- Identity: z -- used in output layers for regression tasks.
Online Backpropagation
For a single sample (x, y), the weight update for layer l is:
W_l <- W_l - eta * dL/dW_l
where eta is the learning rate and dL/dW_l is computed via the chain rule (backpropagation). In the online setting, each update uses exactly one sample, making the gradient estimate noisy but computationally cheap.
Challenges in Online Neural Networks
- Catastrophic forgetting: The network may overwrite knowledge of earlier patterns when adapting to new data.
- Learning rate sensitivity: Too large a rate causes instability; too small a rate prevents adaptation to drift.
- Architecture selection: The number of layers and neurons must be chosen a priori, as online structural adaptation is an open research problem.