advanced

Attention & Transformers

Self-attention, positional encoding, and why Transformers replaced RNNs for everything sequential.

Self-AttentionMulti-Head AttentionPositional EncodingEncoder-DecoderScaled Dot-ProductCross-AttentionGrouped-Query AttentionRoPEFlashAttentionFoundation ModelsScaling LawsMixture of Experts
ML project loop3/6 stages →
DataHypothesis spaceObjectiveOptimizationEvaluationDeployment feedback