Principle:Tensorflow Tfjs Activation Functions

Knowledge Sources	Tensorflow_Tfjs Rectified Linear Units
Domains	Deep_Learning, Neural_Network_Architecture
Last Updated	2026-02-10 06:00 GMT

Overview

Non-linear mathematical functions applied element-wise to layer outputs, enabling neural networks to learn complex decision boundaries beyond linear transformations.

Description

Activation functions introduce non-linearity into neural networks. Without them, any composition of linear layers collapses to a single linear transformation, severely limiting the models representational capacity. TensorFlow.js Layers implements the standard set of activation functions (ReLU, sigmoid, tanh, softmax, etc.) as both standalone classes and string-referenced configurations for use in layer definitions.

Advanced activations like LeakyReLU, PReLU, ELU, ThresholdedReLU, and Softmax are implemented as full Layer subclasses because they carry learnable parameters or require more complex computation than a simple element-wise function.

Usage

Use activation functions when building any neural network architecture. Select based on the layer position and task:

• ReLU for hidden layers (default choice)
• Sigmoid for binary classification output
• Softmax for multi-class classification output
• Tanh for outputs needing range [-1, 1]
• LeakyReLU/PReLU when dying neuron problem is a concern

Theoretical Basis

Common activation functions:

$\text{ReLU}(x)=\max (0,x)$

$\text{Sigmoid}(x)=\frac{1}{1+e^{-x}}$

$\text{Softmax}(x_i)=\frac{e^{x_i}}{\sum _j{e}^{x_j}}$

$\text{ELU}(x)=\{\begin{array}{ll}x& \text{if }x>0\\ \alpha (e^x-1)& \text{if }x\le 0\end{array}$

Related Pages

• Implementation:Tensorflow_Tfjs_Activations
• Implementation:Tensorflow_Tfjs_Advanced_Activations