LSTM & GRU: Gated Memory
The vanishing gradient problem killed long-range learning in vanilla RNNs. The Long Short-Term Memory (LSTM) cell, introduced by Hochreiter & Schmidhuber in 1997, fixed it with two ideas: a separate cell state that acts as a gradient highway, and gates that learn what to remember, forget, and output.
Real-world examples: Before Transformers took over (2017+), LSTMs powered Google Translate, Siri/Alexa speech recognition, handwriting recognition, and most time-series forecasting. They remain strong on small datasets and streaming tasks.
The key innovation: the cell state
An LSTM carries two vectors between steps:
- — the hidden state (the output, the "working memory")
- — the cell state (the long-term memory)
The cell state is updated almost entirely by addition, not repeated matrix multiplication:
That additive path is the gradient highway. Because (a gate near 1, not a fixed weight matrix), gradients can flow across hundreds of steps without vanishing.
Intuition
Think of the cell state as a conveyor belt running straight through the whole sequence. The gates are valves: the forget gate decides what falls off the belt, the input gate decides what new items get placed on it, and the output gate decides what to read off it right now. Information can ride the belt untouched for a long time.
The three gates
Each gate is a small sigmoid layer producing values in — think of them as soft switches, where 0 = "block completely" and 1 = "let everything through." All three look at the previous hidden state and the current input .
1. Forget gate — what to erase from the cell state:
2. Input gate — what new information to write:
3. Output gate — what to expose as the hidden state:
And the state updates:
( is element-wise multiplication; is concatenation.)
| Gate | Question it answers | Output ≈ 0 | Output ≈ 1 |
|---|---|---|---|
| Forget | Keep old memory? | erase it | keep it |
| Input | Store new info? | ignore input | write it |
| Output | Reveal memory now? | stay silent | expose it |
The diagram below animates a single LSTM cell — step through the sequence to watch the cell state accumulate and gate values control information flow.
Step through the sequence to see how the forget gate (red) erases memory, the input gate (purple) writes new content, and the output gate (teal) controls what the hidden state reads. Drag the forget-bias slider to see long-term memory in action.
Worked example: the forget gate in action
Imagine a single cell tracking grammatical subject number (singular vs plural) while reading "The keys to the cabinet ...". At the word "keys" the input gate writes "plural" to the cell. As irrelevant words ("to", "the", "cabinet") stream past, the forget gate stays near 1 (preserving "plural"), so that many steps later the model conjugates "are on the table," not "is." A vanilla RNN would have forgotten the subject by then.
A full LSTM timestep, by hand
Let's run one timestep of a scalar LSTM ( input, hidden) with small explicit numbers so every gate is hand-checkable. The concatenated input is , and each gate has weights and a bias.
Inputs: , , .
Weights (chosen to keep the sums simple):
| Gate | bias | ||
|---|---|---|---|
| Forget | 0 | 0 | |
| Input | 0 | 1 | |
| Candidate | 0 | 1 | 0 |
| Output | 0 | 1 | 0 |
Step 1: pre-activations (each is ):
Step 2: apply the nonlinearities ( on the gates, on the candidate):
Step 3: update the cell state (the additive rule):
Step 4 — emit the hidden state:
So this cell mostly kept its old memory (, carrying of the old forward) and wrote a small amount of new content. These exact numbers are reproduced and verified in the companion notebook.
import numpy as np
def sigmoid(z): return 1 / (1 + np.exp(-z))
# One LSTM step, fully spelled out
def lstm_step(x, h, c, params):
z = np.concatenate([h, x])
f = sigmoid(params['Wf'] @ z + params['bf']) # forget
i = sigmoid(params['Wi'] @ z + params['bi']) # input
c_tilde = np.tanh(params['Wc'] @ z + params['bc']) # candidate
o = sigmoid(params['Wo'] @ z + params['bo']) # output
c = f * c + i * c_tilde # update cell (additive!)
h = o * np.tanh(c) # update hidden
return h, c
Using PyTorch's LSTM
import torch
import torch.nn as nn
lstm = nn.LSTM(input_size=10, hidden_size=20, num_layers=2, batch_first=True)
x = torch.randn(4, 15, 10) # (batch, seq_len, features)
output, (h_n, c_n) = lstm(x)
print(output.shape) # (4, 15, 20) — hidden state per step
print(h_n.shape) # (2, 4, 20) — final hidden, per layer
print(c_n.shape) # (2, 4, 20) — final cell state, per layer
GRU: the leaner cousin
The Gated Recurrent Unit (GRU) (Cho et al., 2014) merges the cell and hidden state into one and uses just two gates — a reset gate and an update gate :
The update gate does the job of the LSTM's forget and input gates at once: it interpolates between keeping the old state and writing the new candidate.
Counting the parameters
Each gate is one linear layer over the concatenation . With input dimension and hidden size , that layer has:
An LSTM has 4 such layers (forget, input, candidate, output), a GRU has 3 (reset, update, candidate):
Concrete count for , . One gate is parameters, so:
The GRU has exactly of the parameters — 25% fewer, because it drops one of the four gate layers. (Both are also verified in the notebook.)
| LSTM | GRU | |
|---|---|---|
| Gates | 3 + candidate (4 layers) | 2 + candidate (3 layers) |
| State vectors | 2 ( and ) | 1 () |
| Parameters (D=10, H=20) | 2,480 | 1,860 (25% fewer) |
| Speed | Slower | Faster |
| Typical edge | Long, complex sequences | Smaller data, faster training |
Which should I use?
There is no universal winner — they perform comparably on most tasks. Start with a GRU (fewer parameters, faster, less prone to overfit on small data). Switch to LSTM if you have lots of data and very long dependencies. Try both and let validation loss decide.
Why gates beat plain RNNs
Differentiate the additive cell update with respect to the previous cell state. Holding the gates fixed (they depend on , but that second-order path is small), the dominant term is strikingly simple:
Chaining this back over steps multiplies the forget gates together:
Compare with the vanilla RNN, where every step multiplies by , so the gradient scales like and a single eigenvalue below 1 makes it vanish exponentially.
Concrete contrast over steps. If the LSTM keeps its forget gate at , the surviving gradient is . A vanilla RNN with an effective per-step factor of gives — about 15× smaller. Push the forget gate to and : nearly two-thirds of the gradient survives 50 steps. That is the gradient highway.
The gradient highway
The additive cell update means the gradient flowing back through the cell state is multiplied by the forget gate (often ≈ 1), not by a weight matrix raised to a power. That is precisely what lets LSTMs and GRUs learn dependencies hundreds of steps long where a vanilla RNN tops out around 10–20.
Exercise
In an LSTM, what does a forget gate value near 1 do to the cell state?
Need a hint?
The forget gate multiplies the previous cell state.
Exercise
Why does the LSTM cell state help with the vanishing gradient problem?
Need a hint?
Look at how the cell state is updated — by addition or by matrix multiplication?
Exercise
How does a GRU differ from an LSTM?
Need a hint?
Compare the number of gates and state vectors.
Common mistakes
- Initializing the forget-gate bias to 0: set to +1 or +2 so the cell defaults to remembering early in training. This is a well-known practical trick.
- Stacking too many layers on small data — 1–2 layers is usually enough; more overfits.
- Reaching for an LSTM when attention is better: for long documents, Transformers (next course) generally beat any RNN. LSTMs shine on small data and streaming.
Key takeaways
- LSTMs add a cell state updated by addition — a gradient highway for long memory.
- Three gates (forget/input/output) learn what to keep, write, and reveal.
- GRU merges them into two gates and one state — fewer parameters, similar performance.
- Initialize the forget-gate bias positive so the cell defaults to remembering.
Related concepts
- BPTT & the Vanishing Gradient: the problem gates were invented to solve
- Self-Attention: what largely superseded LSTMs for long sequences