The Transformer Architecture
Self-attention and multi-head attention are the engine. Now we build the whole car. A Transformer block wraps attention with a feed-forward network, residual connections, and layer normalization — then stacks dozens of these blocks. Understanding the block, and the encoder/decoder/decoder-only families, explains every model from BERT to GPT to T5.
The Transformer block
Each block has two sub-layers, each wrapped in a residual connection and a layer norm:
- Multi-head self-attention: tokens exchange information.
- Position-wise feed-forward network (FFN): each token is independently transformed.
x ──┬─────────────▶(+)──▶ LayerNorm ──┬───────────▶(+)──▶ LayerNorm ──▶ y
│ ▲ │ ▲
└─▶ Multi-Head ─┘ └─▶ FFN ────┘
Attention
(tokens mix here) (each token alone)
Transformer layer, stage by stage
A tensor enters the layer
Token embeddings — with positional information already added — arrive as a matrix of shape [seq_len × d_model]. One row per token, d_model numbers each.
Nothing about this layer is specific to which layer it is. The same block is stacked 12, 48 or 96 times, and every copy sees a tensor of exactly this shape.
Watch the shape readout below as you advance — it is the same at every single step.
tensor shape
[10 × 512] — unchanged since the input, and unchanged at the output.
params at this stage
none — this stage only moves the tensor.
params so far
0 of 3.15M in the layer.
One complete transformer block at d_model = 512, 8 heads, d_ff = 2048. Walk a tensor down it and watch two things: the shape never changes, and the parameter count is dominated by the feed-forward network. Yellow curves are the residual skip connections.
The feed-forward network
A simple two-layer MLP applied to each position independently, usually expanding the dimension 4× then projecting back:
with inner (hidden) dimension , almost always . With that gives . Attention mixes information between tokens; the FFN does the heavy per-token computation. The FFN holds the majority of a Transformer's parameters.
Parameter count of one FFN. is and is , so the weights total (the biases add a negligible ). For :
Two jobs, alternating
A block alternates between communication (attention — tokens look at each other) and computation (FFN — each token thinks on its own). Stacking many blocks lets information mix, get processed, mix again — building progressively richer representations.
Residuals and layer norm: why they matter
- Residual connections () give gradients a direct path to early layers, making it possible to train networks dozens of layers deep — the same trick that made ResNets work.
- Layer normalization stabilizes the scale of activations across the feature dimension, which keeps very deep stacks trainable.
Deriving LayerNorm
For a single token's feature vector , LayerNorm standardizes across the features (not across the batch), then applies a learned scale and shift :
The (tiny, e.g. ) guards against division by zero. After normalizing, every token's features have mean 0 and variance 1; (each a length- vector) then let the network re-scale and re-center if that helps.
Worked example by hand. Take a tiny feature vector
Step 1: mean. .
Step 2: variance. Deviations from the mean are , squared , summing to :
Step 3: normalize (with negligible, divide each deviation by ):
Check: mean ; variance ✓. Step 4 — affine. With (broadcast), . The notebook reproduces these exact numbers.
Why LayerNorm, not BatchNorm, for sequences
BatchNorm normalizes each feature across the batch — it needs a stable batch of comparable examples and breaks when sequences have variable length or the batch is tiny (common in NLP). It also leaks across examples, which is awkward for autoregressive decoding. LayerNorm normalizes across the feature dimension of each token independently, so it is batch-size- and length-agnostic and behaves identically at train and inference time — exactly what sequence models need.
Pre-LN vs Post-LN
The original paper put LayerNorm after the residual add (Post-LN). Modern models (GPT-2 onward) use Pre-LN — normalize before the sub-layer: . Pre-LN trains more stably and needs less learning-rate warmup, so it's now the default.
Counting an encoder layer's parameters
A full encoder layer is the sum of three parts (ignoring the small bias terms):
| Part | Formula | Count at |
|---|---|---|
| Multi-head attention (, each ) | ||
| Feed-forward network | ||
| Two LayerNorms ( each ) |
Note the head count does not change the parameter total: the per-head projections for all heads concatenate back to . Adding the three rows:
The FFN alone is of the parameters and attention is , which is why people say "the FFN is where most of the weights live." A 6-layer encoder is roughly parameters (before embeddings).
Three families
How you mask attention and stack blocks defines the model's purpose:
| Family | Attention | Sees | Good at | Examples |
|---|---|---|---|---|
| Encoder-only | Bidirectional | Whole input at once | Understanding / classification | BERT, RoBERTa |
| Decoder-only | Causal (masked) | Only past tokens | Generation | GPT, LLaMA, Claude |
| Encoder–decoder | Both | Input + generated output | Seq2seq (translation) | T5, original Transformer |
Encoder vs decoder layers
An encoder layer is exactly what we counted above: bidirectional self-attention → FFN, each with residual + LayerNorm. A decoder layer has three sub-layers instead of two:
- Masked (causal) self-attention: the same self-attention, but a triangular mask sets all "future" scores to before the softmax, so position can attend only to positions . This is what makes generation autoregressive: while predicting token the model cannot peek at .
- Cross-attention: queries come from the decoder, keys/values from the encoder's output. This is how the decoder "looks at" the source.
- Feed-forward network: same per-token MLP.
The original Transformer translates with two stacks. The encoder reads the source sentence bidirectionally. The decoder generates the target one token at a time, mixing the masked self-attention (what it has written so far) with cross-attention (the source it is translating).
import torch
import torch.nn as nn
# A full encoder block in a few lines
encoder_layer = nn.TransformerEncoderLayer(
d_model=512, nhead=8, dim_feedforward=2048,
norm_first=True, batch_first=True, # norm_first=True → Pre-LN
)
encoder = nn.TransformerEncoder(encoder_layer, num_layers=6)
x = torch.randn(2, 20, 512) # (batch, seq, d_model)
out = encoder(x)
print(out.shape) # (2, 20, 512)
End-to-end data flow (decoder-only / GPT)
- Tokenize text into integer IDs. Modern LLMs use Byte-Pair Encoding (BPE) or similar subword tokenizers — common words become a single token while rare words split into familiar subword pieces, eliminating out-of-vocabulary (
[UNK]) tokens. See the Text Preprocessing lesson for full details on BPE and WordPiece. - Embed each ID into a vector; add positional encoding.
- Pass through Transformer blocks (with causal self-attention).
- Final LayerNorm, then a linear projection to vocabulary size.
- Softmax → probability of the next token. Sample, append, repeat.
import torch.nn as nn
class TinyGPT(nn.Module):
def __init__(self, vocab, d_model=256, n_heads=4, n_layers=4, max_len=128):
super().__init__()
self.tok = nn.Embedding(vocab, d_model)
self.pos = nn.Embedding(max_len, d_model)
layer = nn.TransformerEncoderLayer(d_model, n_heads, 4 * d_model,
norm_first=True, batch_first=True)
self.blocks = nn.TransformerEncoder(layer, n_layers)
self.ln = nn.LayerNorm(d_model)
self.head = nn.Linear(d_model, vocab)
def forward(self, idx):
T = idx.size(1)
pos = torch.arange(T, device=idx.device)
x = self.tok(idx) + self.pos(pos)
mask = nn.Transformer.generate_square_subsequent_mask(T).to(idx.device)
x = self.blocks(x, mask=mask) # causal mask → autoregressive
return self.head(self.ln(x)) # logits over vocab
Scale is the story
Transformers improved largely by getting bigger — more layers, wider dimensions, more data. Empirically, loss falls as a smooth power law in parameters, data, and compute (the scaling laws, covered in Foundation Models & Scaling Laws). GPT-3 (175B params) and beyond are "just" this same block, stacked very high and trained on enormous corpora.
Quadratic cost is the ceiling
Self-attention is in sequence length. Doubling context quadruples attention compute and memory. Extending context length (FlashAttention, sparse/linear attention, RoPE scaling) is one of the most active areas in the field — the next lesson covers these techniques in detail.
Exercise
In a Transformer block, the position-wise feed-forward network (FFN) operates how?
Need a hint?
Does it mix information between tokens or process each one alone?
Exercise
Why are residual connections important in deep Transformer stacks?
Need a hint?
Think about how gradients reach early layers.
Exercise
GPT-style language models use which Transformer family?
Need a hint?
They generate text one token at a time and must not see the future.
Common mistakes
- Wrong mask for the family: encoders use no causal mask; decoders require one. Mixing them up either leaks the future or cripples understanding.
- Dropping the residual or LayerNorm: deep stacks become untrainable without them.
- Assuming bigger always wins: scale needs proportional data; an under-fed huge model overfits.
Key takeaways
- A block alternates attention (mix tokens) and a feed-forward net (per-token compute).
- Residuals + LayerNorm make very deep stacks trainable.
- Encoder-only (BERT), decoder-only (GPT), and encoder–decoder (T5) suit different tasks.
- Transformers improved largely by scaling params, data, and compute.
Related concepts
- Multi-Head Attention & Positional Encoding: the attention and positioning inside each block
- Self-Attention: the core mechanism it's built on