LSTM & GRUOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

LSTM & GRU: Gated Memory

The vanishing gradient problem killed long-range learning in vanilla RNNs. The Long Short-Term Memory (LSTM) cell, introduced by Hochreiter & Schmidhuber in 1997, fixed it with two ideas: a separate cell state that acts as a gradient highway, and gates that learn what to remember, forget, and output.

Real-world examples: Before Transformers took over (2017+), LSTMs powered Google Translate, Siri/Alexa speech recognition, handwriting recognition, and most time-series forecasting. They remain strong on small datasets and streaming tasks.

The key innovation: the cell state

An LSTM carries two vectors between steps:

  • ht\mathbf{h}_t — the hidden state (the output, the "working memory")
  • ct\mathbf{c}_t — the cell state (the long-term memory)

The cell state is updated almost entirely by addition, not repeated matrix multiplication:

ct=ft⊙ct−1+it⊙c~t\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t

That additive path is the gradient highway. Because ∂ct/∂ct−1≈ft\partial \mathbf{c}_t / \partial \mathbf{c}_{t-1} \approx \mathbf{f}_t (a gate near 1, not a fixed weight matrix), gradients can flow across hundreds of steps without vanishing.

Intuition

Think of the cell state as a conveyor belt running straight through the whole sequence. The gates are valves: the forget gate decides what falls off the belt, the input gate decides what new items get placed on it, and the output gate decides what to read off it right now. Information can ride the belt untouched for a long time.

The three gates

Each gate is a small sigmoid layer producing values in (0,1)(0, 1) — think of them as soft switches, where 0 = "block completely" and 1 = "let everything through." All three look at the previous hidden state ht−1\mathbf{h}_{t-1} and the current input xt\mathbf{x}_t.

1. Forget gate — what to erase from the cell state:

ft=σ(Wf[ht−1,xt]+bf)\mathbf{f}_t = \sigma(W_f[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_f)

2. Input gate — what new information to write:

it=σ(Wi[ht−1,xt]+bi)\mathbf{i}_t = \sigma(W_i[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_i) c~t=tanh⁡(Wc[ht−1,xt]+bc)(candidate values)\tilde{\mathbf{c}}_t = \tanh(W_c[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_c) \quad\text{(candidate values)}

3. Output gate — what to expose as the hidden state:

ot=σ(Wo[ht−1,xt]+bo)\mathbf{o}_t = \sigma(W_o[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_o)

And the state updates:

ct=ft⊙ct−1+it⊙c~tht=ot⊙tanh⁡(ct)\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t \qquad \mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t)

(⊙\odot is element-wise multiplication; [⋅,⋅][\cdot,\cdot] is concatenation.)

GateQuestion it answersOutput ≈ 0Output ≈ 1
Forget ft\mathbf{f}_tKeep old memory?erase itkeep it
Input it\mathbf{i}_tStore new info?ignore inputwrite it
Output ot\mathbf{o}_tReveal memory now?stay silentexpose it

The diagram below animates a single LSTM cell — step through the sequence to watch the cell state accumulate and gate values control information flow.

Cell C_t-0.225Forget f_t0.380Input i_t0.427Output o_t0.404~C cand.-0.527Hidden h_t-0.089x_t =-0.977t=1t=8
step t1
f_t forget0.380
i_t input0.427
o_t output0.404
C_t cell-0.225
h_t hidden-0.089

Step through the sequence to see how the forget gate (red) erases memory, the input gate (purple) writes new content, and the output gate (teal) controls what the hidden state reads. Drag the forget-bias slider to see long-term memory in action.

Worked example: the forget gate in action

Imagine a single cell tracking grammatical subject number (singular vs plural) while reading "The keys to the cabinet ...". At the word "keys" the input gate writes "plural" to the cell. As irrelevant words ("to", "the", "cabinet") stream past, the forget gate stays near 1 (preserving "plural"), so that many steps later the model conjugates "are on the table," not "is." A vanilla RNN would have forgotten the subject by then.

A full LSTM timestep, by hand

Let's run one timestep of a scalar LSTM (D=1D = 1 input, H=1H = 1 hidden) with small explicit numbers so every gate is hand-checkable. The concatenated input is z=[ ht−1, xt ]z = [\,h_{t-1},\, x_t\,], and each gate has weights [Wh,Wx][W_h, W_x] and a bias.

Inputs: ht−1=0h_{t-1} = 0, xt=1x_t = 1, ct−1=2c_{t-1} = 2.

Weights (chosen to keep the sums simple):

GateWhW_hWxW_xbias
Forget ff00+2+2
Input ii01−1-1
Candidate c~\tilde{c}010
Output oo010

Step 1: pre-activations (each is Whht−1+Wxxt+bW_h h_{t-1} + W_x x_t + b):

zf=0⋅0+0⋅1+2=2,zi=0⋅0+1⋅1−1=0,zc~=1,zo=1z_f = 0\cdot 0 + 0\cdot 1 + 2 = 2,\quad z_i = 0\cdot 0 + 1\cdot 1 - 1 = 0,\quad z_{\tilde c} = 1,\quad z_o = 1

Step 2: apply the nonlinearities (σ\sigma on the gates, tanh⁡\tanh on the candidate):

ft=σ(2)=11+e−2≈0.881it=σ(0)=0.5c~t=tanh⁡(1)≈0.762ot=σ(1)≈0.731\begin{aligned} f_t &= \sigma(2) = \tfrac{1}{1+e^{-2}} \approx 0.881 \\ i_t &= \sigma(0) = 0.5 \\ \tilde{c}_t &= \tanh(1) \approx 0.762 \\ o_t &= \sigma(1) \approx 0.731 \end{aligned}

Step 3: update the cell state (the additive rule):

ct=ft ct−1+it c~t=0.881×2+0.5×0.762=1.762+0.381=2.142c_t = f_t\, c_{t-1} + i_t\, \tilde{c}_t = 0.881\times 2 + 0.5\times 0.762 = 1.762 + 0.381 = \boxed{2.142}

Step 4 — emit the hidden state:

ht=ottanh⁡(ct)=0.731×tanh⁡(2.142)=0.731×0.973=0.711h_t = o_t \tanh(c_t) = 0.731 \times \tanh(2.142) = 0.731 \times 0.973 = \boxed{0.711}

So this cell mostly kept its old memory (ft≈0.88f_t \approx 0.88, carrying 1.761.76 of the old 2.02.0 forward) and wrote a small amount of new content. These exact numbers are reproduced and verified in the companion notebook.

Python
import numpy as np

def sigmoid(z): return 1 / (1 + np.exp(-z))

# One LSTM step, fully spelled out
def lstm_step(x, h, c, params):
    z = np.concatenate([h, x])
    f = sigmoid(params['Wf'] @ z + params['bf'])     # forget
    i = sigmoid(params['Wi'] @ z + params['bi'])     # input
    c_tilde = np.tanh(params['Wc'] @ z + params['bc'])  # candidate
    o = sigmoid(params['Wo'] @ z + params['bo'])     # output
    c = f * c + i * c_tilde                           # update cell (additive!)
    h = o * np.tanh(c)                                # update hidden
    return h, c

Using PyTorch's LSTM

Python
import torch
import torch.nn as nn

lstm = nn.LSTM(input_size=10, hidden_size=20, num_layers=2, batch_first=True)
x = torch.randn(4, 15, 10)                # (batch, seq_len, features)
output, (h_n, c_n) = lstm(x)

print(output.shape)   # (4, 15, 20) — hidden state per step
print(h_n.shape)      # (2, 4, 20) — final hidden, per layer
print(c_n.shape)      # (2, 4, 20) — final cell state, per layer

GRU: the leaner cousin

The Gated Recurrent Unit (GRU) (Cho et al., 2014) merges the cell and hidden state into one and uses just two gates — a reset gate rt\mathbf{r}_t and an update gate zt\mathbf{z}_t:

zt=σ(Wz[ht−1,xt])rt=σ(Wr[ht−1,xt])\mathbf{z}_t = \sigma(W_z[\mathbf{h}_{t-1}, \mathbf{x}_t]) \qquad \mathbf{r}_t = \sigma(W_r[\mathbf{h}_{t-1}, \mathbf{x}_t]) h~t=tanh⁡(Wh[rt⊙ht−1,xt])\tilde{\mathbf{h}}_t = \tanh(W_h[\mathbf{r}_t \odot \mathbf{h}_{t-1}, \mathbf{x}_t]) ht=(1−zt)⊙ht−1+zt⊙h~t\mathbf{h}_t = (1 - \mathbf{z}_t)\odot \mathbf{h}_{t-1} + \mathbf{z}_t \odot \tilde{\mathbf{h}}_t

The update gate zt\mathbf{z}_t does the job of the LSTM's forget and input gates at once: it interpolates between keeping the old state and writing the new candidate.

Counting the parameters

Each gate is one linear layer over the concatenation [ht−1,xt][\mathbf{h}_{t-1}, \mathbf{x}_t]. With input dimension DD and hidden size HH, that layer has:

H×D⏟Wx+H×H⏟Wh+H⏟bias  =  H(D+H+1) parameters per gate.\underbrace{H\times D}_{W_x} + \underbrace{H\times H}_{W_h} + \underbrace{H}_{\text{bias}} \;=\; H(D + H + 1) \text{ parameters per gate.}

An LSTM has 4 such layers (forget, input, candidate, output), a GRU has 3 (reset, update, candidate):

PLSTM=4 H(D+H+1),PGRU=3 H(D+H+1).P_{\text{LSTM}} = 4\,H(D + H + 1), \qquad P_{\text{GRU}} = 3\,H(D + H + 1).

Concrete count for D=10D = 10, H=20H = 20. One gate is 20(10+20+1)=20×31=62020(10 + 20 + 1) = 20\times 31 = 620 parameters, so:

PLSTM=4×620=2,480,PGRU=3×620=1,860.P_{\text{LSTM}} = 4\times 620 = 2{,}480, \qquad P_{\text{GRU}} = 3\times 620 = 1{,}860.

The GRU has exactly 1860/2480=0.751860/2480 = 0.75 of the parameters — 25% fewer, because it drops one of the four gate layers. (Both are also verified in the notebook.)

LSTMGRU
Gates3 + candidate (4 layers)2 + candidate (3 layers)
State vectors2 (h\mathbf{h} and c\mathbf{c})1 (h\mathbf{h})
Parameters (D=10, H=20)2,4801,860 (25% fewer)
SpeedSlowerFaster
Typical edgeLong, complex sequencesSmaller data, faster training

Which should I use?

There is no universal winner — they perform comparably on most tasks. Start with a GRU (fewer parameters, faster, less prone to overfit on small data). Switch to LSTM if you have lots of data and very long dependencies. Try both and let validation loss decide.

Why gates beat plain RNNs

Differentiate the additive cell update with respect to the previous cell state. Holding the gates fixed (they depend on ht−1\mathbf{h}_{t-1}, but that second-order path is small), the dominant term is strikingly simple:

ct=ft⊙ct−1+it⊙c~t⟹∂ct∂ct−1≈ft.\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t \quad\Longrightarrow\quad \frac{\partial \mathbf{c}_t}{\partial \mathbf{c}_{t-1}} \approx \mathbf{f}_t.

Chaining this back over TT steps multiplies the forget gates together:

∂cT∂c0≈∏t=1Tft.\frac{\partial \mathbf{c}_T}{\partial \mathbf{c}_0} \approx \prod_{t=1}^{T} \mathbf{f}_t.

Compare with the vanilla RNN, where every step multiplies by Whh⊤ diag(tanh⁡′)W_{hh}^\top \,\mathrm{diag}(\tanh'), so the gradient scales like (Whh)T\big(W_{hh}\big)^{T} and a single eigenvalue below 1 makes it vanish exponentially.

Concrete contrast over T=50T = 50 steps. If the LSTM keeps its forget gate at f=0.95f = 0.95, the surviving gradient is 0.9550≈0.0770.95^{50} \approx 0.077. A vanilla RNN with an effective per-step factor of 0.90.9 gives 0.950≈0.0050.9^{50} \approx 0.005 — about 15× smaller. Push the forget gate to f=0.99f = 0.99 and 0.9950≈0.610.99^{50} \approx 0.61: nearly two-thirds of the gradient survives 50 steps. That is the gradient highway.

The gradient highway

The additive cell update ct=ft⊙ct−1+…\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \dots means the gradient flowing back through the cell state is multiplied by the forget gate (often ≈ 1), not by a weight matrix raised to a power. That is precisely what lets LSTMs and GRUs learn dependencies hundreds of steps long where a vanilla RNN tops out around 10–20.

Exercise

In an LSTM, what does a forget gate value near 1 do to the cell state?

Need a hint?

The forget gate multiplies the previous cell state.

Exercise

Why does the LSTM cell state help with the vanishing gradient problem?

Need a hint?

Look at how the cell state is updated — by addition or by matrix multiplication?

Exercise

How does a GRU differ from an LSTM?

Need a hint?

Compare the number of gates and state vectors.

Common mistakes

  1. Initializing the forget-gate bias to 0: set bf\mathbf{b}_f to +1 or +2 so the cell defaults to remembering early in training. This is a well-known practical trick.
  2. Stacking too many layers on small data — 1–2 layers is usually enough; more overfits.
  3. Reaching for an LSTM when attention is better: for long documents, Transformers (next course) generally beat any RNN. LSTMs shine on small data and streaming.

Key takeaways

  • LSTMs add a cell state updated by addition — a gradient highway for long memory.
  • Three gates (forget/input/output) learn what to keep, write, and reveal.
  • GRU merges them into two gates and one state — fewer parameters, similar performance.
  • Initialize the forget-gate bias positive so the cell defaults to remembering.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
LSTM & GRUNeural NetworksCalculus for MLBPTT & the Vanishing Grad…Self-AttentionDeep Learning for Time Se…Backpropagation Through T…Partially Observable MDPs…Vanishing & Exploding Gra…