Attention & Transformers
Self-attention, positional encoding, and why Transformers replaced RNNs for everything sequential.
7 lessons
01Self-Attention
The mechanism that replaced recurrence — how every token looks at every other token at once.
Multi-Head Attention & Positional Encoding
Many attention heads in parallel, and how a permutation-invariant model learns word order.
The Transformer Architecture
Assembling attention into full blocks — residuals, layer norm, FFN, and the encoder/decoder families.
Modern Attention Mechanisms
Cross-attention, grouped-query attention, RoPE, FlashAttention, and the optimizations that make large language models practical.
Foundation Models & Scaling Laws
How GPT-class models are pretrained, the Chinchilla compute-optimal recipe, what the context window really costs, and where SFT/RLHF fits in.
Mixture of Experts
Scaling models without scaling cost per token — sparse expert layers, top-k gating, load balancing, and why MoE decouples a model's parameter count from its compute.
Transformers Quiz
Test your understanding of self-attention, multi-head attention, positional encoding, the Transformer block, modern attention variants, scaling laws, and mixture of experts.