Principle:LaurentMazare Tch rs Recurrent Neural Network

Knowledge Sources	LaurentMazare_Tch_rs Hochreiter & Schmidhuber, 1997 Cho et al., 2014
Domains	Deep Learning, Sequence Modeling, Natural Language Processing
Last Updated	2026-02-08 00:00 GMT

Overview

Recurrent neural networks process sequential data by maintaining a hidden state that is updated at each time step, with gated variants (LSTM, GRU) designed to capture long-range dependencies.

Description

Recurrent neural networks (RNNs) are a class of neural architectures designed for sequential data where the order of elements matters. At each time step, an RNN takes the current input and its previous hidden state, producing an updated hidden state and optionally an output. This hidden state serves as the network's memory, accumulating information from all previously seen elements in the sequence.

The basic (vanilla) RNN suffers from the vanishing gradient problem: gradients diminish exponentially as they are backpropagated through many time steps, making it difficult to learn dependencies between distant elements. Two gated architectures address this limitation:

Long Short-Term Memory (LSTM) introduces a separate cell state that flows through the network with minimal transformation, acting as a "conveyor belt" for information. Three gates control information flow:

• Forget gate -- Decides what information to discard from the cell state
• Input gate -- Decides what new information to store in the cell state
• Output gate -- Decides what information from the cell state to output as the hidden state

Gated Recurrent Unit (GRU) simplifies the LSTM architecture by merging the cell state and hidden state into a single state vector. It uses two gates:

• Reset gate -- Controls how much of the previous hidden state to forget when computing the candidate new state
• Update gate -- Controls the interpolation between the previous hidden state and the candidate new state

Both LSTM and GRU can be stacked into multiple layers, where the output sequence from one layer becomes the input to the next. Bidirectional variants process the sequence in both forward and reverse directions, concatenating the resulting hidden states.

Usage

Apply recurrent architectures when:

• Processing time series, audio, or text data where temporal order is significant
• The input and/or output are variable-length sequences
• Long-range dependencies must be captured (prefer LSTM or GRU over vanilla RNN)
• Building seq2seq models, language models, or sequence classifiers

Theoretical Basis

Vanilla RNN

At each time step $t$, given input $x_t$ and previous hidden state $h_{t-1}$:

$h_t=\tanh (W_{ih}{x}_t+b_{ih}+W_{hh}{h}_{t-1}+b_{hh})$

The vanishing gradient arises because $\frac{\partial h_T}{\partial h_1}=\prod _{t=2}^T\frac{\partial h_t}{\partial h_{t-1}}$, which tends toward zero when the spectral radius of the Jacobian is less than 1.

LSTM

The LSTM introduces cell state $c_t$ alongside hidden state $h_t$:

Forget gate: $f_t=\sigma (W_f{x}_t+U_f{h}_{t-1}+b_f)$

Input gate: $i_t=\sigma (W_i{x}_t+U_i{h}_{t-1}+b_i)$

Cell candidate: ${\tilde{c}}_t=\tanh (W_c{x}_t+U_c{h}_{t-1}+b_c)$

Cell state update: $c_t=f_t\odot c_t{}_{-1}+i_t\odot {\tilde{c}}_t$

Output gate: $o_t=\sigma (W_o{x}_t+U_o{h}_{t-1}+b_o)$

Hidden state: $h_t=o_t\odot \tanh (c_t)$

The cell state update $c_t=f_t\odot c_{t-1}+\dots$ creates an additive gradient path, preventing vanishing gradients.

GRU

The GRU uses fewer parameters than LSTM:

Reset gate: $r_t=\sigma (W_r{x}_t+U_r{h}_{t-1}+b_r)$

Update gate: $z_t=\sigma (W_z{x}_t+U_z{h}_{t-1}+b_z)$

Candidate state: ${\tilde{h}}_t=\tanh (W_h{x}_t+U_h(r_t\odot h_{t-1})+b_h)$

Hidden state: $h_t=(1-z_t)\odot h_{t-1}+z_t\odot {\tilde{h}}_t$

The update gate $z_t$ controls interpolation between the old and new states, allowing the GRU to either retain previous information or incorporate new information.

Bidirectional Processing

A bidirectional RNN runs two separate recurrent networks -- one processing the sequence forward and one backward. The outputs are concatenated:

$h_t^{\text{bi}}=[\overrightarrow{h_t};\overleftarrow{h_t}]$

This provides each position with context from both past and future elements.

Related Pages

• Implementation:LaurentMazare_Tch_rs_RNN