Principle:Sktime Pytorch forecasting xLSTM Architecture

Knowledge Sources	xLSTMTime: Long-term Time Series Forecasting With xLSTM pytorch-forecasting
Domains	Time_Series, Forecasting, Deep_Learning, Recurrent_Neural_Networks
Last Updated	2026-02-08 09:00 GMT

Overview

Extended LSTM (xLSTM) architecture for long-term time series forecasting, combining two novel LSTM variants -- mLSTM (matrix LSTM with matrix memory and exponential gating) and sLSTM (stabilized LSTM with normalized exponential gating) -- with series decomposition and normalization for robust multi-horizon predictions.

Description

xLSTMTime extends the classical LSTM by introducing two complementary cell designs that address the limited memory capacity and gradient stability issues of standard LSTMs:

mLSTM (Matrix LSTM): Replaces the scalar cell state with a matrix memory mechanism. The cell computes query, key, and value vectors from the input, forms an outer-product key-value interaction matrix, and accumulates this into the cell state via exponential gating. The hidden state is derived by multiplying the output gate with the tanh of the cell state scaled by a normalized accumulator. This gives the cell a richer, higher-capacity memory compared to standard scalar LSTM states. The mLSTM cell maintains three states: hidden state $h$, cell state $c$, and normalizer state $n$.

sLSTM (Stabilized LSTM): Retains the scalar memory structure of a traditional LSTM but introduces normalized exponential gating for the input and forget gates. Instead of sigmoid activations, each gate uses a centered exponential function followed by normalization so that the input and forget gates sum to a stable partition. This prevents the vanishing/exploding gate problem while maintaining the simplicity of scalar-memory LSTMs. Layer normalization is applied throughout the cell for training stability.

xLSTMTime model architecture:

1. Series decomposition: The input encoder sequence is decomposed into trend and seasonal components using a moving-average kernel of configurable size.
2. Concatenation and projection: Trend and seasonal components are concatenated along the feature dimension and linearly projected to the hidden size.
3. Batch normalization: Applied across the hidden dimension for stable training.
4. Recurrent processing: The projected sequence is fed through a stacked mLSTM or sLSTM network (selectable via the xlstm_type parameter). Each network consists of multiple recurrent layers with residual connections, layer normalization, and dropout.
5. Output projection: The final hidden state is linearly projected to the forecast horizon, followed by instance normalization.

Both the mLSTM and sLSTM variants are organized in a three-tier hierarchy: Cell (single time-step computation), Layer (multiple stacked cells over a sequence), and Network (layers plus a fully connected output head).

Usage

Use xLSTMTime when a recurrent architecture is preferred over a transformer for time series forecasting, particularly when: (1) the series exhibits strong sequential dependencies that benefit from recurrent inductive biases, (2) computational resources are limited relative to the sequence length (recurrent models avoid quadratic attention cost), and (3) the time series contains both trend and seasonal components. Choose mlstm for higher memory capacity tasks and slstm for simpler series where training stability is paramount.

Theoretical Basis

mLSTM cell update equations:

$q_t=W_q{x}_t,k_t=\frac{W_k{x}_t}{\sqrt{d}},v_t=W_v{x}_t$

$i_t=\sigma (W_i{x}_t),f_t=\sigma (W_f{x}_t),o_t=\sigma (W_o{x}_t)$

$c_t=f_t\odot c_{t-1}+i_t\odot \sum (k_t\otimes v_t)$

$n_t=f_t\odot n_{t-1}+i_t\odot k_t$

$h_t=o_t\odot \tanh (c_t\odot \frac{n_t}{\Vert n_t\Vert +ϵ})$

where $\otimes$ denotes the outer product, $\odot$ denotes element-wise multiplication, and $n_t$ is the normalizer state that stabilizes the memory readout.

sLSTM cell update equations:

${\text{gates}}_x=W_{input}{x}_t,{\text{gates}}_h=W_{hidden}{h}_{t-1}$

$[i,f,g,o]={\text{gates}}_x+{\text{gates}}_h$

Normalized exponential gating:

${\tilde{i}}_t=\frac{\exp ({\overline{i}}_t)}{\sum \exp ({\overline{i}}_t)+ϵ},{\tilde{f}}_t=\frac{\exp ({\overline{f}}_t)}{\sum \exp ({\overline{f}}_t)+ϵ}$

$i_t^{*}=\frac{{\tilde{i}}_t}{{\tilde{i}}_t+{\tilde{f}}_t+ϵ},f_t^{*}=\frac{{\tilde{f}}_t}{{\tilde{i}}_t+{\tilde{f}}_t+ϵ}$

$c_t=f_t^{*}\odot c_{t-1}+i_t^{*}\odot \tanh (g_t)$

$h_t=\sigma (o_t)\odot \tanh (c_t)$

where the centering and clamping of pre-gate values (within [-5, 5]) ensures numerical stability.

Series decomposition:

${\text{trend}}_t={\text{AvgPool}}_k(x_t),{\text{seasonal}}_t=x_t-{\text{trend}}_t$

Overall xLSTMTime forward pass:

seasonal, trend = SeriesDecomposition(encoder_input)
x = Linear(concat(trend, seasonal))
x = BatchNorm(x)
output, hidden = xLSTM_Network(x)        # mLSTM or sLSTM
prediction = InstanceNorm(Linear(output))

Key hyperparameters:

• xlstm_type -- "slstm" or "mlstm" variant selection
• hidden_size -- recurrent hidden dimension
• num_layers -- number of stacked recurrent layers
• decomposition_kernel -- moving-average kernel size (default: 25)
• dropout -- recurrent dropout rate (default: 0.1)

Related Pages

Implemented By

• Implementation:Sktime_Pytorch_forecasting_xLSTMTime
• Implementation:Sktime_Pytorch_forecasting_mLSTMCell
• Implementation:Sktime_Pytorch_forecasting_mLSTMLayer
• Implementation:Sktime_Pytorch_forecasting_mLSTMNetwork
• Implementation:Sktime_Pytorch_forecasting_sLSTMCell
• Implementation:Sktime_Pytorch_forecasting_sLSTMLayer
• Implementation:Sktime_Pytorch_forecasting_sLSTMNetwork