Recurrent Neural NetworksOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Recurrent Neural Networks

A feed-forward network maps a fixed-size input to a fixed-size output. But language, audio, stock prices, and DNA are sequences — their length varies and their order matters. A Recurrent Neural Network (RNN) processes a sequence one element at a time, carrying a hidden state that summarizes everything it has seen so far. Reusing the same weights at every step is a hypothesis-space choice — weight sharing across time, the third of the four architectures in the inductive-bias story.

Deep dive
The ML Project Loop
Includes runnable notebook

The forecasting thread

Here the site's running daily-demand problem (from the project-loop hub) gets its sequence-model treatment. The linear baseline hand-built lag features; an ARIMA model assumed linearity. An RNN instead learns what to carry forward in its hidden state — the same demand problem, now with a hypothesis space that discovers its own temporal features rather than being handed them.

Real-world examples: Language modeling, machine translation, speech recognition, time-series forecasting, sentiment analysis, music generation.

The core idea: a loop

A feed-forward layer computes y=f(Wx)\mathbf{y} = f(W\mathbf{x}) once. An RNN applies the same layer repeatedly, feeding its own previous output back in as extra input:

ht=tanh⁡ ⁣(Wxh xt+Whh ht−1+bh)\mathbf{h}_t = \tanh\!\left(W_{xh}\,\mathbf{x}_t + W_{hh}\,\mathbf{h}_{t-1} + \mathbf{b}_h\right) yt=Why ht+by\mathbf{y}_t = W_{hy}\,\mathbf{h}_t + \mathbf{b}_y
  • xt\mathbf{x}_t — input at time step tt (e.g. the embedding of the tt-th word), a DD-vector
  • ht\mathbf{h}_t — hidden state at time tt, the network's "memory", an HH-vector
  • ht−1\mathbf{h}_{t-1} — hidden state from the previous step
  • WxhW_{xh} (H×DH\times D), WhhW_{hh} (H×HH\times H), WhyW_{hy} (O×HO\times H) — weight matrices, shared across all time steps

Intuition

The hidden state is like reading a book and keeping a running summary in your head. Each new sentence (xt\mathbf{x}_t) updates your understanding (ht\mathbf{h}_t) based on what you already knew (ht−1\mathbf{h}_{t-1}). You never re-read the whole book — the summary carries the context forward.

Why the hidden state is memory

Unroll the recurrence by substituting ht−1\mathbf{h}_{t-1}, then ht−2\mathbf{h}_{t-2}, and so on:

ht=tanh⁡ ⁣(Wxhxt+Whh tanh⁡(Wxhxt−1+Whhht−2+bh)⏟ht−1+bh)\mathbf{h}_t = \tanh\!\Big(W_{xh}\mathbf{x}_t + W_{hh}\,\underbrace{\tanh(W_{xh}\mathbf{x}_{t-1} + W_{hh}\mathbf{h}_{t-2} + \mathbf{b}_h)}_{\mathbf{h}_{t-1}} + \mathbf{b}_h\Big)

Every past input x1,…,xt\mathbf{x}_1,\dots,\mathbf{x}_t has left a fingerprint inside ht\mathbf{h}_t. There is no separate storage buffer — the single fixed-size vector ht\mathbf{h}_t is a lossy compression of the entire prefix. That is exactly why we call it memory: change an earlier input and ht\mathbf{h}_t changes too.

Unrolling the loop

The recurrence is easier to see "unrolled" in time — the same cell drawn once per step, passing h\mathbf{h} to the right:

Code
x₁      x₂      x₃      x₄
 │       │       │       │
 ▼       ▼       ▼       ▼
[h]──▶──[h]──▶──[h]──▶──[h]──▶  hₜ
 │       │       │       │
 ▼       ▼       ▼       ▼
y₁      y₂      y₃      y₄

The diagram below unrolls the RNN — the same WhW_h reappears at every step, and the hidden state carries information rightward:

RNN unrolled over 5 timesteps

W_h← same W_h reused at every step →t=00.8,0.2t=1-0.3,0.9t=20.5,-0.4t=30.1,0.7t=4-0.6,0.3output y_thidden h_tinput x_t
t0
x_t[0.8, 0.2]
h_t[0.22, 0.07, 0.31]
y_t-0.017

The same weight matrix $W_h$ connects every hidden state to the next — this is weight sharing. Step through the sequence to see the hidden state evolve, carrying context forward.

Crucially, every box is the same set of weights. This parameter sharing is why an RNN can handle a sequence of any length with a fixed number of parameters, and why a pattern learned at position 2 transfers to position 200.

Weight sharing matters

PropertyFeed-forward netRNN
Input lengthFixedVariable
Parameters grow with length?YesNo (shared)
Sees order of inputs?NoYes
Has memory of past?NoYes (hidden state)

How many parameters? Counting an RNN cell

Because the weights are shared, the parameter count is independent of sequence length. For input size DD, hidden size HH, output size OO:

TensorShapeCount
WxhW_{xh}H×DH\times DHDHD
WhhW_{hh}H×HH\times HH2H^2
bh\mathbf{b}_hHHHH
WhyW_{hy}O×HO\times HOHOH
by\mathbf{b}_yOOOO
total=HD+H2+H⏟recurrent cell+OH+O⏟output head\text{total} = \underbrace{HD + H^2 + H}_{\text{recurrent cell}} + \underbrace{OH + O}_{\text{output head}}

Worked count for D=3,  H=4,  O=2D=3,\;H=4,\;O=2:

4⋅3⏟12+42⏟16+4⏟4+2⋅4⏟8+2⏟2=42\underbrace{4\cdot 3}_{12} + \underbrace{4^2}_{16} + \underbrace{4}_{4} + \underbrace{2\cdot 4}_{8} + \underbrace{2}_{2} = 42

The recurrent cell alone (the part reused at every step) is HD+H2+H=12+16+4=32HD + H^2 + H = 12 + 16 + 4 = 32 parameters. Process a length-3 or a length-3000 sequence. It is still these same 32 numbers.

Worked forward pass: two timesteps by hand

Let's run the full recurrence with D=2D=2 input, H=2H=2 hidden, O=1O=1 output, and tiny explicit weights:

Wxh=[0.5−0.30.10.4],Whh=[0.2000.2],bh=[00],Why=[1−1],by=0W_{xh}=\begin{bmatrix}0.5 & -0.3\\ 0.1 & 0.4\end{bmatrix},\quad W_{hh}=\begin{bmatrix}0.2 & 0\\ 0 & 0.2\end{bmatrix},\quad \mathbf{b}_h=\begin{bmatrix}0\\0\end{bmatrix},\quad W_{hy}=\begin{bmatrix}1 & -1\end{bmatrix},\quad b_y=0

Start from h0=[00]\mathbf{h}_0 = \begin{bmatrix}0\\0\end{bmatrix} and feed x1=[12]\mathbf{x}_1=\begin{bmatrix}1\\2\end{bmatrix}, then x2=[−11]\mathbf{x}_2=\begin{bmatrix}-1\\1\end{bmatrix}.

Step 1. Since h0=0\mathbf{h}_0=\mathbf{0}, the Whhh0W_{hh}\mathbf{h}_0 term vanishes:

Wxhx1=[0.5(1)−0.3(2)0.1(1)+0.4(2)]=[−0.10.9]W_{xh}\mathbf{x}_1=\begin{bmatrix}0.5(1)-0.3(2)\\ 0.1(1)+0.4(2)\end{bmatrix}=\begin{bmatrix}-0.1\\ 0.9\end{bmatrix} h1=tanh⁡ ⁣[−0.10.9]=[tanh⁡(−0.1)tanh⁡(0.9)]=[−0.09970.7163]\mathbf{h}_1=\tanh\!\begin{bmatrix}-0.1\\ 0.9\end{bmatrix}=\begin{bmatrix}\tanh(-0.1)\\ \tanh(0.9)\end{bmatrix}=\begin{bmatrix}-0.0997\\ 0.7163\end{bmatrix} y1=Whyh1=1(−0.0997)+(−1)(0.7163)=−0.8160y_1 = W_{hy}\mathbf{h}_1 = 1(-0.0997) + (-1)(0.7163) = -0.8160

Step 2. Now the previous hidden state feeds back in. First the two contributions to the pre-activation:

Wxhx2=[0.5(−1)−0.3(1)0.1(−1)+0.4(1)]=[−0.80.3],Whhh1=0.2 h1=[−0.01990.1433]W_{xh}\mathbf{x}_2=\begin{bmatrix}0.5(-1)-0.3(1)\\ 0.1(-1)+0.4(1)\end{bmatrix}=\begin{bmatrix}-0.8\\ 0.3\end{bmatrix},\qquad W_{hh}\mathbf{h}_1=0.2\,\mathbf{h}_1=\begin{bmatrix}-0.0199\\ 0.1433\end{bmatrix} pre2=[−0.80.3]+[−0.01990.1433]=[−0.81990.4433]\text{pre}_2 = \begin{bmatrix}-0.8\\ 0.3\end{bmatrix}+\begin{bmatrix}-0.0199\\ 0.1433\end{bmatrix}=\begin{bmatrix}-0.8199\\ 0.4433\end{bmatrix} h2=tanh⁡ ⁣[−0.81990.4433]=[−0.67500.4163]\mathbf{h}_2=\tanh\!\begin{bmatrix}-0.8199\\ 0.4433\end{bmatrix}=\begin{bmatrix}-0.6750\\ 0.4163\end{bmatrix} y2=1(−0.6750)+(−1)(0.4163)=−1.0914y_2 = 1(-0.6750) + (-1)(0.4163) = -1.0914

Notice the Whhh1W_{hh}\mathbf{h}_1 term: h2\mathbf{h}_2 depends on x1\mathbf{x}_1 through h1\mathbf{h}_1. That feedback path is the memory channel. The notebook recomputes every number above and asserts they match.

Worked example: a 1-D hidden state

To see memory accumulate over more steps, drop to a scalar: Wxh=0.5W_{xh}=0.5, Whh=0.9W_{hh}=0.9, bh=0b_h=0, h0=0h_0=0, inputs x=[1,1,1]x = [1, 1, 1].

h1=tanh⁡(0.5⋅1+0.9⋅0)=tanh⁡(0.5)=0.462h_1 = \tanh(0.5\cdot 1 + 0.9\cdot 0) = \tanh(0.5) = 0.462 h2=tanh⁡(0.5⋅1+0.9⋅0.462)=tanh⁡(0.916)=0.724h_2 = \tanh(0.5\cdot 1 + 0.9\cdot 0.462) = \tanh(0.916) = 0.724 h3=tanh⁡(0.5⋅1+0.9⋅0.724)=tanh⁡(1.152)=0.818h_3 = \tanh(0.5\cdot 1 + 0.9\cdot 0.724) = \tanh(1.152) = 0.818

The hidden state accumulates — each step builds on the last. The tanh⁡\tanh keeps it bounded in (−1,1)(-1, 1) so it cannot blow up within a step.

Sequence-to-what? The four shapes

RNNs flex to different tasks depending on how many inputs and outputs you wire up:

PatternInputs → OutputsExample
one-to-many1 → NImage → caption
many-to-oneN → 1Review → sentiment label
many-to-many (aligned)N → NPart-of-speech tagging
many-to-many (seq2seq)N → MTranslation (encoder–decoder)

From scratch in code

The same recurrence, with a small random RNN over a length-5 sequence:

Python
import numpy as np

def rnn_step(x, h, W_xh, W_hh, b_h):
    return np.tanh(W_xh @ x + W_hh @ h + b_h)

# A tiny RNN: 3-dim input, 4-dim hidden state
np.random.seed(0)
W_xh = np.random.randn(4, 3) * 0.1
W_hh = np.random.randn(4, 4) * 0.1
b_h  = np.zeros(4)

h = np.zeros(4)                       # initial hidden state
sequence = [np.random.randn(3) for _ in range(5)]

for t, x in enumerate(sequence):
    h = rnn_step(x, h, W_xh, W_hh, b_h)
    print(f"t={t}  h = {np.round(h, 3)}")

Using PyTorch's built-in RNN

Python
import torch
import torch.nn as nn

rnn = nn.RNN(input_size=3, hidden_size=4, batch_first=True)

# (batch=2, seq_len=5, features=3)
x = torch.randn(2, 5, 3)
output, h_n = rnn(x)

print(output.shape)  # (2, 5, 4) — hidden state at every time step
print(h_n.shape)     # (1, 2, 4) — final hidden state only

output vs h_n

output gives you the hidden state at every step (use for tagging / seq2seq). h_n is just the final hidden state (use for classification, e.g. sentiment). They share the last step: output[:, -1, :] == h_n[0].

Strengths and limits

Strengths: handle variable-length input, model order and context, compact (shared weights), natural fit for streaming/online data.

Limits:

  1. Sequential by nature: step tt needs ht−1\mathbf{h}_{t-1}, so training cannot parallelize across time.
  2. Short memory: vanilla RNNs struggle to connect events far apart (next lesson).
  3. Vanishing/exploding gradients: repeated multiplication by WhhW_{hh} during training destabilizes learning.

Vanilla RNNs forget fast

In practice a plain RNN reliably remembers only ~10–20 steps. The "vanishing gradient" problem (covered next) is the reason LSTMs and GRUs were invented.

Exercise

What role does the hidden state h_t play in a recurrent neural network?

Need a hint?

Think about what gets passed from one time step to the next.

Exercise

Why can an RNN process sequences of any length with a fixed number of parameters?

Need a hint?

Consider what is reused at every time step.

Exercise

Classifying the sentiment of a product review (a sequence of words) into one label is which RNN pattern?

Need a hint?

Count the inputs and outputs.

Common mistakes

  1. Forgetting to reset the hidden state between independent sequences — context leaks across unrelated examples.
  2. Not detaching/truncating the hidden state across long sequences — backprop grows unbounded in memory.
  3. Feeding raw token IDs instead of embeddings — RNNs expect continuous vectors, not category indices.

Key takeaways

  • An RNN carries a hidden state that summarizes the sequence seen so far.
  • Weight sharing across time steps handles variable-length input with fixed parameters.
  • Output wiring (one/many in/out) adapts it to tagging, classification, or seq2seq.
  • Vanilla RNNs reliably remember only ~10–20 steps.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Recurrent Neural NetworksNeural NetworksCalculus for MLChain Rule and Backpropag…Linear RegressionARIMA ModelsBPTT & the Vanishing Grad…What is a Neuron?Self-AttentionInductive Bias & Equivari…The ML Project Loop