The Transformer ArchitectureOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

The Transformer Architecture

Self-attention and multi-head attention are the engine. Now we build the whole car. A Transformer block wraps attention with a feed-forward network, residual connections, and layer normalization — then stacks dozens of these blocks. Understanding the block, and the encoder/decoder/decoder-only families, explains every model from BERT to GPT to T5.

The Transformer block

Each block has two sub-layers, each wrapped in a residual connection and a layer norm:

  1. Multi-head self-attention: tokens exchange information.
  2. Position-wise feed-forward network (FFN): each token is independently transformed.
z=LayerNorm(x+MultiHead(x))\mathbf{z} = \text{LayerNorm}\big(\mathbf{x} + \text{MultiHead}(\mathbf{x})\big) y=LayerNorm(z+FFN(z))\mathbf{y} = \text{LayerNorm}\big(\mathbf{z} + \text{FFN}(\mathbf{z})\big)
Code
   x ──┬─────────────▶(+)──▶ LayerNorm ──┬───────────▶(+)──▶ LayerNorm ──▶ y
       │               ▲                 │             ▲
       └─▶ Multi-Head ─┘                 └─▶  FFN  ────┘
            Attention
        (tokens mix here)            (each token alone)

Transformer layer, stage by stage

Forward pass · one layerstep 1 of 6
A tensor enters the layer

Token embeddings — with positional information already added — arrive as a matrix of shape [seq_len × d_model]. One row per token, d_model numbers each.

Nothing about this layer is specific to which layer it is. The same block is stacked 12, 48 or 96 times, and every copy sees a tensor of exactly this shape.

Watch the shape readout below as you advance — it is the same at every single step.

Inputencoder · d_model 512 · 8 heads
Input EmbeddingMulti-Head AttentionAdd & NormFeed-Forward (FFN)Add & NormOutputresidualresidual
tensor shape

[10 × 512] — unchanged since the input, and unchanged at the output.

params at this stage

none — this stage only moves the tensor.

params so far

0 of 3.15M in the layer.

learned sub-layerresidual add + LayerNormskip connection

One complete transformer block at d_model = 512, 8 heads, d_ff = 2048. Walk a tensor down it and watch two things: the shape never changes, and the parameter count is dominated by the feed-forward network. Yellow curves are the residual skip connections.

The feed-forward network

A simple two-layer MLP applied to each position independently, usually expanding the dimension 4× then projecting back:

FFN(x)=W2 ReLU(W1x+b1)+b2\text{FFN}(\mathbf{x}) = W_2\,\text{ReLU}(W_1\mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2

with inner (hidden) dimension dffd_{ff}, almost always dff=4 dmodeld_{ff} = 4\,d_{\text{model}}. With dmodel=512d_{\text{model}} = 512 that gives dff=2048d_{ff} = 2048. Attention mixes information between tokens; the FFN does the heavy per-token computation. The FFN holds the majority of a Transformer's parameters.

Parameter count of one FFN. W1W_1 is dmodel×dffd_{\text{model}} \times d_{ff} and W2W_2 is dff×dmodeld_{ff} \times d_{\text{model}}, so the weights total 2 dmodel dff2\,d_{\text{model}}\,d_{ff} (the biases b1,b2\mathbf{b}_1, \mathbf{b}_2 add a negligible dff+dmodeld_{ff} + d_{\text{model}}). For dmodel=512, dff=2048d_{\text{model}}=512,\ d_{ff}=2048:

2⋅512⋅2048=2,097,152≈2.1M weights.2 \cdot 512 \cdot 2048 = 2{,}097{,}152 \approx 2.1\text{M weights}.

Two jobs, alternating

A block alternates between communication (attention — tokens look at each other) and computation (FFN — each token thinks on its own). Stacking many blocks lets information mix, get processed, mix again — building progressively richer representations.

Residuals and layer norm: why they matter

  • Residual connections (x+Sublayer(x)\mathbf{x} + \text{Sublayer}(\mathbf{x})) give gradients a direct path to early layers, making it possible to train networks dozens of layers deep — the same trick that made ResNets work.
  • Layer normalization stabilizes the scale of activations across the feature dimension, which keeps very deep stacks trainable.

Deriving LayerNorm

For a single token's feature vector x∈Rd\mathbf{x} \in \mathbb{R}^{d}, LayerNorm standardizes across the dd features (not across the batch), then applies a learned scale γ\boldsymbol{\gamma} and shift β\boldsymbol{\beta}:

μ=1d∑i=1dxi,σ2=1d∑i=1d(xi−μ)2,\mu = \frac{1}{d}\sum_{i=1}^{d} x_i, \qquad \sigma^2 = \frac{1}{d}\sum_{i=1}^{d} (x_i - \mu)^2, y=γ⊙x−μσ2+ϵ+β.\mathbf{y} = \boldsymbol{\gamma} \odot \frac{\mathbf{x} - \mu}{\sqrt{\sigma^2 + \epsilon}} + \boldsymbol{\beta}.

The ϵ\epsilon (tiny, e.g. 10−510^{-5}) guards against division by zero. After normalizing, every token's features have mean 0 and variance 1; γ,β\boldsymbol{\gamma}, \boldsymbol{\beta} (each a length-dd vector) then let the network re-scale and re-center if that helps.

Worked example by hand. Take a tiny d=8d = 8 feature vector

x=[ 2, 4, 4, 4, 5, 5, 7, 9 ].\mathbf{x} = [\,2,\ 4,\ 4,\ 4,\ 5,\ 5,\ 7,\ 9\,].

Step 1: mean. μ=2+4+4+4+5+5+7+98=408=5\mu = \dfrac{2+4+4+4+5+5+7+9}{8} = \dfrac{40}{8} = 5.

Step 2: variance. Deviations from the mean are [−3,−1,−1,−1,0,0,2,4][-3,-1,-1,-1,0,0,2,4], squared [9,1,1,1,0,0,4,16][9,1,1,1,0,0,4,16], summing to 3232:

σ2=328=4,σ2=2.\sigma^2 = \frac{32}{8} = 4, \qquad \sqrt{\sigma^2} = 2.

Step 3: normalize (with ϵ\epsilon negligible, divide each deviation by σ=2\sigma = 2):

x^=x−52=[ −1.5, −0.5, −0.5, −0.5, 0, 0, 1.0, 2.0 ].\hat{\mathbf{x}} = \frac{\mathbf{x} - 5}{2} = [\,-1.5,\ -0.5,\ -0.5,\ -0.5,\ 0,\ 0,\ 1.0,\ 2.0\,].

Check: mean =−1.5−0.5−0.5−0.5+0+0+1.0+2.08=08=0= \frac{-1.5-0.5-0.5-0.5+0+0+1.0+2.0}{8} = \frac{0}{8} = 0; variance =2.25+0.25+0.25+0.25+0+0+1+48=88=1= \frac{2.25+0.25+0.25+0.25+0+0+1+4}{8} = \frac{8}{8} = 1 ✓. Step 4 — affine. With γ=2, β=1\boldsymbol{\gamma} = 2,\ \boldsymbol{\beta} = 1 (broadcast), y=2x^+1=[−2,0,0,0,1,1,3,5]\mathbf{y} = 2\hat{\mathbf{x}} + 1 = [-2, 0, 0, 0, 1, 1, 3, 5]. The notebook reproduces these exact numbers.

Why LayerNorm, not BatchNorm, for sequences

BatchNorm normalizes each feature across the batch — it needs a stable batch of comparable examples and breaks when sequences have variable length or the batch is tiny (common in NLP). It also leaks across examples, which is awkward for autoregressive decoding. LayerNorm normalizes across the feature dimension of each token independently, so it is batch-size- and length-agnostic and behaves identically at train and inference time — exactly what sequence models need.

Pre-LN vs Post-LN

The original paper put LayerNorm after the residual add (Post-LN). Modern models (GPT-2 onward) use Pre-LN — normalize before the sub-layer: x+Attn(LN(x))\mathbf{x} + \text{Attn}(\text{LN}(\mathbf{x})). Pre-LN trains more stably and needs less learning-rate warmup, so it's now the default.

Counting an encoder layer's parameters

A full encoder layer is the sum of three parts (ignoring the small bias terms):

PartFormulaCount at dmodel=512, dff=2048, h=8d_{\text{model}}=512,\ d_{ff}=2048,\ h=8
Multi-head attention (WQ,WK,WV,WOW_Q, W_K, W_V, W_O, each d×dd\times d)4 dmodel24\,d_{\text{model}}^24⋅5122=1,048,5764 \cdot 512^2 = 1{,}048{,}576
Feed-forward network2 dmodel dff2\,d_{\text{model}}\,d_{ff}2⋅512⋅2048=2,097,1522 \cdot 512 \cdot 2048 = 2{,}097{,}152
Two LayerNorms (γ,β\boldsymbol{\gamma}, \boldsymbol{\beta} each dd)4 dmodel4\,d_{\text{model}}4⋅512=2,0484 \cdot 512 = 2{,}048

Note the head count hh does not change the parameter total: the per-head projections d×dhd \times \tfrac{d}{h} for all hh heads concatenate back to d×dd \times d. Adding the three rows:

1,048,576+2,097,152+2,048=3,147,776≈3.15M per layer.1{,}048{,}576 + 2{,}097{,}152 + 2{,}048 = 3{,}147{,}776 \approx 3.15\text{M per layer}.

The FFN alone is ≈66.6%\approx 66.6\% of the parameters and attention is ≈33.3%\approx 33.3\%, which is why people say "the FFN is where most of the weights live." A 6-layer encoder is roughly 6×3.15M≈19M6 \times 3.15\text{M} \approx 19\text{M} parameters (before embeddings).

Three families

How you mask attention and stack blocks defines the model's purpose:

FamilyAttentionSeesGood atExamples
Encoder-onlyBidirectionalWhole input at onceUnderstanding / classificationBERT, RoBERTa
Decoder-onlyCausal (masked)Only past tokensGenerationGPT, LLaMA, Claude
Encoder–decoderBothInput + generated outputSeq2seq (translation)T5, original Transformer

Encoder vs decoder layers

An encoder layer is exactly what we counted above: bidirectional self-attention → FFN, each with residual + LayerNorm. A decoder layer has three sub-layers instead of two:

  1. Masked (causal) self-attention: the same self-attention, but a triangular mask sets all "future" scores to −∞-\infty before the softmax, so position tt can attend only to positions ≤t\le t. This is what makes generation autoregressive: while predicting token tt the model cannot peek at t+1t+1.
  2. Cross-attention: queries come from the decoder, keys/values from the encoder's output. This is how the decoder "looks at" the source.
  3. Feed-forward network: same per-token MLP.

The original Transformer translates with two stacks. The encoder reads the source sentence bidirectionally. The decoder generates the target one token at a time, mixing the masked self-attention (what it has written so far) with cross-attention (the source it is translating).

Python
import torch
import torch.nn as nn

# A full encoder block in a few lines
encoder_layer = nn.TransformerEncoderLayer(
    d_model=512, nhead=8, dim_feedforward=2048,
    norm_first=True, batch_first=True,           # norm_first=True → Pre-LN
)
encoder = nn.TransformerEncoder(encoder_layer, num_layers=6)

x = torch.randn(2, 20, 512)                      # (batch, seq, d_model)
out = encoder(x)
print(out.shape)                                 # (2, 20, 512)

End-to-end data flow (decoder-only / GPT)

  1. Tokenize text into integer IDs. Modern LLMs use Byte-Pair Encoding (BPE) or similar subword tokenizers — common words become a single token while rare words split into familiar subword pieces, eliminating out-of-vocabulary ([UNK]) tokens. See the Text Preprocessing lesson for full details on BPE and WordPiece.
  2. Embed each ID into a dmodeld_{\text{model}} vector; add positional encoding.
  3. Pass through NN Transformer blocks (with causal self-attention).
  4. Final LayerNorm, then a linear projection to vocabulary size.
  5. Softmax → probability of the next token. Sample, append, repeat.
Python
import torch.nn as nn

class TinyGPT(nn.Module):
    def __init__(self, vocab, d_model=256, n_heads=4, n_layers=4, max_len=128):
        super().__init__()
        self.tok = nn.Embedding(vocab, d_model)
        self.pos = nn.Embedding(max_len, d_model)
        layer = nn.TransformerEncoderLayer(d_model, n_heads, 4 * d_model,
                                           norm_first=True, batch_first=True)
        self.blocks = nn.TransformerEncoder(layer, n_layers)
        self.ln = nn.LayerNorm(d_model)
        self.head = nn.Linear(d_model, vocab)

    def forward(self, idx):
        T = idx.size(1)
        pos = torch.arange(T, device=idx.device)
        x = self.tok(idx) + self.pos(pos)
        mask = nn.Transformer.generate_square_subsequent_mask(T).to(idx.device)
        x = self.blocks(x, mask=mask)             # causal mask → autoregressive
        return self.head(self.ln(x))              # logits over vocab

Scale is the story

Transformers improved largely by getting bigger — more layers, wider dimensions, more data. Empirically, loss falls as a smooth power law in parameters, data, and compute (the scaling laws, covered in Foundation Models & Scaling Laws). GPT-3 (175B params) and beyond are "just" this same block, stacked very high and trained on enormous corpora.

Quadratic cost is the ceiling

Self-attention is O(n2)O(n^2) in sequence length. Doubling context quadruples attention compute and memory. Extending context length (FlashAttention, sparse/linear attention, RoPE scaling) is one of the most active areas in the field — the next lesson covers these techniques in detail.

Exercise

In a Transformer block, the position-wise feed-forward network (FFN) operates how?

Need a hint?

Does it mix information between tokens or process each one alone?

Exercise

Why are residual connections important in deep Transformer stacks?

Need a hint?

Think about how gradients reach early layers.

Exercise

GPT-style language models use which Transformer family?

Need a hint?

They generate text one token at a time and must not see the future.

Common mistakes

  1. Wrong mask for the family: encoders use no causal mask; decoders require one. Mixing them up either leaks the future or cripples understanding.
  2. Dropping the residual or LayerNorm: deep stacks become untrainable without them.
  3. Assuming bigger always wins: scale needs proportional data; an under-fed huge model overfits.

Key takeaways

  • A block alternates attention (mix tokens) and a feed-forward net (per-token compute).
  • Residuals + LayerNorm make very deep stacks trainable.
  • Encoder-only (BERT), decoder-only (GPT), and encoder–decoder (T5) suit different tasks.
  • Transformers improved largely by scaling params, data, and compute.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
The Transformer ArchitectureRecurrent Neural NetworksCalculus for MLBatchNorm & DropoutMulti-Head Attention & Po…Text PreprocessingFoundation Models & Scali…Modern Attention Mechanis…Self-AttentionBatch Normalization: Forw…GroupNorm & InstanceNorm:…