Implementation:Online ml River NeuralNet Activations
| Knowledge Sources | |
|---|---|
| Domains | Neural_Networks, Activation_Functions |
| Last Updated | 2026-02-08 16:00 GMT |
Overview
Activation functions for neural networks providing non-linear transformations with efficient forward and gradient computations.
Description
This module implements three core activation functions as stateless classes. Each activation provides two static methods: apply() for forward computation and gradient() for backpropagation. ReLU (Rectified Linear Unit) outputs max(0, z), with gradient 1 where z > 0. Sigmoid computes 1/(1+exp(-z)) with gradient s(1-s). Identity is a pass-through (no transformation) with constant gradient 1. All methods are pure functions without side effects, operating on numpy arrays.
Usage
Use ReLU for hidden layers as it prevents vanishing gradients and enables sparse activation. Use Sigmoid for binary outputs or when you need bounded (0,1) activations. Use Identity for regression output layers or as a pass-through. Activations are specified when constructing neural networks - typically ReLU for hidden layers and Identity for regression outputs.
Code Reference
Source Location
- Repository: Online_ml_River
- File: river/neural_net/activations.py
Signature
class Activation:
@abc.abstractstaticmethod
def apply(self, z):
"""Apply the activation function to a layer output z."""
@abc.abstractstaticmethod
def gradient(self, z):
"""Return the gradient with respect to a layer output z."""
class ReLU(Activation):
@staticmethod
def apply(z):
a = np.copy(z)
a[a < 0] = 0
return a
@staticmethod
def gradient(z):
a = np.zeros_like(z, dtype=z.dtype)
a[z > 0] = 1
return a
class Sigmoid(Activation):
@staticmethod
def apply(z):
return 1 / (1 + np.exp(-z))
@staticmethod
def gradient(z):
s = Sigmoid.apply(z)
return s * (1 - s)
class Identity(Activation):
@staticmethod
def apply(z):
return np.copy(z)
@staticmethod
def gradient(z):
return np.ones_like(z, dtype=z.dtype)
Import
from river import neural_net as nn
# Then use nn.activations.ReLU, nn.activations.Sigmoid, etc.
I/O Contract
Methods
| Method | Input | Output | Description |
|---|---|---|---|
| apply | z: ndarray | ndarray | Apply activation function element-wise |
| gradient | z: ndarray | ndarray | Compute gradient element-wise |
Activation Functions
| Function | Formula | Gradient | Use Case |
|---|---|---|---|
| ReLU | max(0, z) | 1 if z > 0 else 0 | Hidden layers |
| Sigmoid | 1/(1+e^-z) | σ(z)(1-σ(z)) | Binary output |
| Identity | z | 1 | Regression output |
Usage Examples
from river import neural_net as nn
from river import preprocessing as pp
from river import datasets
from river import evaluate
from river import metrics
from river import optim
# Using activations in MLP
model = (
pp.StandardScaler() |
nn.MLPRegressor(
hidden_dims=(10, 5), # 2 hidden layers
activations=(
nn.activations.ReLU, # Input -> Hidden1
nn.activations.ReLU, # Hidden1 -> Hidden2
nn.activations.Identity # Hidden2 -> Output
),
optimizer=optim.SGD(1e-3),
seed=42
)
)
dataset = datasets.TrumpApproval()
metric = metrics.MAE()
evaluate.progressive_val_score(dataset, model, metric)
# Direct usage (numpy arrays)
import numpy as np
z = np.array([-2, -1, 0, 1, 2])
# ReLU
nn.activations.ReLU.apply(z)
# array([0, 0, 0, 1, 2])
nn.activations.ReLU.gradient(z)
# array([0, 0, 0, 1, 1])
# Sigmoid
nn.activations.Sigmoid.apply(z)
# array([0.119, 0.268, 0.5, 0.731, 0.880])
# Identity
nn.activations.Identity.apply(z)
# array([-2, -1, 0, 1, 2])