Recurrent Neural Networks
A feed-forward network maps a fixed-size input to a fixed-size output. But language, audio, stock prices, and DNA are sequences — their length varies and their order matters. A Recurrent Neural Network (RNN) processes a sequence one element at a time, carrying a hidden state that summarizes everything it has seen so far. Reusing the same weights at every step is a hypothesis-space choice — weight sharing across time, the third of the four architectures in the inductive-bias story.
The forecasting thread
Here the site's running daily-demand problem (from the project-loop hub) gets its sequence-model treatment. The linear baseline hand-built lag features; an ARIMA model assumed linearity. An RNN instead learns what to carry forward in its hidden state — the same demand problem, now with a hypothesis space that discovers its own temporal features rather than being handed them.
Real-world examples: Language modeling, machine translation, speech recognition, time-series forecasting, sentiment analysis, music generation.
The core idea: a loop
A feed-forward layer computes once. An RNN applies the same layer repeatedly, feeding its own previous output back in as extra input:
- — input at time step (e.g. the embedding of the -th word), a -vector
- — hidden state at time , the network's "memory", an -vector
- — hidden state from the previous step
- (), (), () — weight matrices, shared across all time steps
Intuition
The hidden state is like reading a book and keeping a running summary in your head. Each new sentence () updates your understanding () based on what you already knew (). You never re-read the whole book — the summary carries the context forward.
Why the hidden state is memory
Unroll the recurrence by substituting , then , and so on:
Every past input has left a fingerprint inside . There is no separate storage buffer — the single fixed-size vector is a lossy compression of the entire prefix. That is exactly why we call it memory: change an earlier input and changes too.
Unrolling the loop
The recurrence is easier to see "unrolled" in time — the same cell drawn once per step, passing to the right:
x₁ x₂ x₃ x₄
│ │ │ │
▼ ▼ ▼ ▼
[h]──▶──[h]──▶──[h]──▶──[h]──▶ hₜ
│ │ │ │
▼ ▼ ▼ ▼
y₁ y₂ y₃ y₄
The diagram below unrolls the RNN — the same reappears at every step, and the hidden state carries information rightward:
RNN unrolled over 5 timesteps
The same weight matrix $W_h$ connects every hidden state to the next — this is weight sharing. Step through the sequence to see the hidden state evolve, carrying context forward.
Crucially, every box is the same set of weights. This parameter sharing is why an RNN can handle a sequence of any length with a fixed number of parameters, and why a pattern learned at position 2 transfers to position 200.
Weight sharing matters
| Property | Feed-forward net | RNN |
|---|---|---|
| Input length | Fixed | Variable |
| Parameters grow with length? | Yes | No (shared) |
| Sees order of inputs? | No | Yes |
| Has memory of past? | No | Yes (hidden state) |
How many parameters? Counting an RNN cell
Because the weights are shared, the parameter count is independent of sequence length. For input size , hidden size , output size :
| Tensor | Shape | Count |
|---|---|---|
Worked count for :
The recurrent cell alone (the part reused at every step) is parameters. Process a length-3 or a length-3000 sequence. It is still these same 32 numbers.
Worked forward pass: two timesteps by hand
Let's run the full recurrence with input, hidden, output, and tiny explicit weights:
Start from and feed , then .
Step 1. Since , the term vanishes:
Step 2. Now the previous hidden state feeds back in. First the two contributions to the pre-activation:
Notice the term: depends on through . That feedback path is the memory channel. The notebook recomputes every number above and asserts they match.
Worked example: a 1-D hidden state
To see memory accumulate over more steps, drop to a scalar: , , , , inputs .
The hidden state accumulates — each step builds on the last. The keeps it bounded in so it cannot blow up within a step.
Sequence-to-what? The four shapes
RNNs flex to different tasks depending on how many inputs and outputs you wire up:
| Pattern | Inputs → Outputs | Example |
|---|---|---|
| one-to-many | 1 → N | Image → caption |
| many-to-one | N → 1 | Review → sentiment label |
| many-to-many (aligned) | N → N | Part-of-speech tagging |
| many-to-many (seq2seq) | N → M | Translation (encoder–decoder) |
From scratch in code
The same recurrence, with a small random RNN over a length-5 sequence:
import numpy as np
def rnn_step(x, h, W_xh, W_hh, b_h):
return np.tanh(W_xh @ x + W_hh @ h + b_h)
# A tiny RNN: 3-dim input, 4-dim hidden state
np.random.seed(0)
W_xh = np.random.randn(4, 3) * 0.1
W_hh = np.random.randn(4, 4) * 0.1
b_h = np.zeros(4)
h = np.zeros(4) # initial hidden state
sequence = [np.random.randn(3) for _ in range(5)]
for t, x in enumerate(sequence):
h = rnn_step(x, h, W_xh, W_hh, b_h)
print(f"t={t} h = {np.round(h, 3)}")
Using PyTorch's built-in RNN
import torch
import torch.nn as nn
rnn = nn.RNN(input_size=3, hidden_size=4, batch_first=True)
# (batch=2, seq_len=5, features=3)
x = torch.randn(2, 5, 3)
output, h_n = rnn(x)
print(output.shape) # (2, 5, 4) — hidden state at every time step
print(h_n.shape) # (1, 2, 4) — final hidden state only
output vs h_n
output gives you the hidden state at every step (use for tagging / seq2seq). h_n is just the final hidden state (use for classification, e.g. sentiment). They share the last step: output[:, -1, :] == h_n[0].
Strengths and limits
Strengths: handle variable-length input, model order and context, compact (shared weights), natural fit for streaming/online data.
Limits:
- Sequential by nature: step needs , so training cannot parallelize across time.
- Short memory: vanilla RNNs struggle to connect events far apart (next lesson).
- Vanishing/exploding gradients: repeated multiplication by during training destabilizes learning.
Vanilla RNNs forget fast
In practice a plain RNN reliably remembers only ~10–20 steps. The "vanishing gradient" problem (covered next) is the reason LSTMs and GRUs were invented.
Exercise
What role does the hidden state h_t play in a recurrent neural network?
Need a hint?
Think about what gets passed from one time step to the next.
Exercise
Why can an RNN process sequences of any length with a fixed number of parameters?
Need a hint?
Consider what is reused at every time step.
Exercise
Classifying the sentiment of a product review (a sequence of words) into one label is which RNN pattern?
Need a hint?
Count the inputs and outputs.
Common mistakes
- Forgetting to reset the hidden state between independent sequences — context leaks across unrelated examples.
- Not detaching/truncating the hidden state across long sequences — backprop grows unbounded in memory.
- Feeding raw token IDs instead of embeddings — RNNs expect continuous vectors, not category indices.
Key takeaways
- An RNN carries a hidden state that summarizes the sequence seen so far.
- Weight sharing across time steps handles variable-length input with fixed parameters.
- Output wiring (one/many in/out) adapts it to tagging, classification, or seq2seq.
- Vanilla RNNs reliably remember only ~10–20 steps.
Related concepts
- BPTT & the Vanishing Gradient: why long-range memory is hard
- What is a Neuron?: the unit unrolled through time
- Self-Attention: the mechanism that replaced recurrence
- Inductive Bias & Equivariance: the RNN as weight sharing across time, and where its recency bias breaks