Implementation:Tensorflow Tfjs Advanced Activations

Knowledge Sources	Tensorflow_Tfjs TensorFlow.js API
Domains	Deep_Learning, Layers_API
Last Updated	2026-02-10 06:00 GMT

Overview

The advanced_activations.ts module implements activation functions as full Layer subclasses, as opposed to the simpler function-based activations in activations.ts. These layer-based activations can have trainable parameters (like PReLU's learnable alpha), configurable hyperparameters, and can be used as standalone layers in a model graph.

The module provides six advanced activation layers: ReLU (with optional max value), LeakyReLU, PReLU (with trainable alpha), ELU, ThresholdedReLU, and Softmax (as a layer with configurable axis and mask support). All classes extend Layer and are registered with serialization.registerClass().

Code Reference

Source Location

tfjs-layers/src/layers/advanced_activations.ts (View on GitHub)

Key Signatures

export class ReLU extends Layer {
  static className = 'ReLU';
  maxValue: number;
  constructor(args?: ReLULayerArgs);
  call(inputs: Tensor | Tensor[], kwargs: Kwargs): Tensor | Tensor[];
}

export class LeakyReLU extends Layer {
  static className = 'LeakyReLU';
  readonly alpha: number; // default 0.3
  constructor(args?: LeakyReLULayerArgs);
  call(inputs: Tensor | Tensor[], kwargs: Kwargs): Tensor | Tensor[];
}

export class PReLU extends Layer {
  static className = 'PReLU';
  constructor(args?: PReLULayerArgs);
  build(inputShape: Shape | Shape[]): void;
  call(inputs: Tensor | Tensor[], kwargs: Kwargs): Tensor | Tensor[];
}

export class ELU extends Layer {
  static className = 'ELU';
  readonly alpha: number; // default 1.0
  constructor(args?: ELULayerArgs);
  call(inputs: Tensor | Tensor[], kwargs: Kwargs): Tensor | Tensor[];
}

export class ThresholdedReLU extends Layer {
  static className = 'ThresholdedReLU';
  readonly theta: number; // default 1.0
  constructor(args?: ThresholdedReLULayerArgs);
  call(inputs: Tensor | Tensor[], kwargs: Kwargs): Tensor | Tensor[];
}

export class Softmax extends Layer {
  static className = 'Softmax';
  readonly axis: number | number[];
  constructor(args?: SoftmaxLayerArgs);
  call(inputs: Tensor | Tensor[], kwargs: Kwargs): Tensor | Tensor[];
}

Import

import { LeakyReLU, ELU, PReLU, ThresholdedReLU, Softmax, ReLU } from './layers/advanced_activations';

Layer Details

Layer	Parameters	Formula
ReLU	maxValue (optional)	min(maxValue, max(0, x))
LeakyReLU	alpha (default 0.3)	x >= 0 ? x : alpha * x
PReLU	alphaInitializer, alphaRegularizer, alphaConstraint, sharedAxes	x >= 0 ? x : alpha * x (alpha is trainable)
ELU	alpha (default 1.0, only 1.0 supported)	x >= 0 ? x : alpha * (exp(x) - 1)
ThresholdedReLU	theta (default 1.0)	x > theta ? x : 0
Softmax	axis (default -1)	Normalized exponentials, supports masks

I/O Contract

Input

All layers accept tensors of arbitrary shape. PReLU requires the build() method to be called first (done automatically when added to a model) to initialize the learnable alpha weight. Softmax supports an optional mask in kwargs to mask out positions before softmax computation.

Output

All layers return tensors of the same shape as the input. Output values depend on the specific activation function applied element-wise.

PReLU Build

During build(), PReLU creates a trainable alpha weight with shape derived from the input shape (excluding batch dimension). If sharedAxes is specified, those dimensions in alpha are set to 1 for parameter sharing.

Usage Examples

import * as tf from '@tensorflow/tfjs';

// LeakyReLU with custom alpha
const model = tf.sequential();
model.add(tf.layers.dense({ inputShape: [10], units: 32 }));
model.add(tf.layers.leakyReLU({ alpha: 0.1 }));

// PReLU with shared axes for conv features
model.add(tf.layers.conv2d({ filters: 64, kernelSize: 3, inputShape: [28, 28, 1] }));
model.add(tf.layers.prelu({ sharedAxes: [1, 2] }));

// Softmax layer with specific axis
const softmaxLayer = tf.layers.softmax({ axis: -1 });
const output = softmaxLayer.apply(tf.randomNormal([2, 10]));

// ThresholdedReLU
const threshLayer = tf.layers.thresholdedReLU({ theta: 0.5 });

Related Pages

• Tensorflow_Tfjs_Activations - Function-based activations (non-layer, used as activation parameters)
• Tensorflow_Tfjs_Exports_Layers - Public API factory functions (tf.layers.leakyReLU, tf.layers.prelu, etc.)
• Tensorflow_Tfjs_Layer_Variables - LayerVariable used by PReLU for its trainable alpha weight