Optimization for ML
How models actually learn — gradient descent variants, convexity theory, and constrained optimization.
6 lessons
01Gradient Descent Variants
SGD, momentum, RMSprop, and Adam — how the optimizer zoo evolved and when to use each.
Convex Optimization
Why convexity is the gold standard — Jensen's inequality, global minima guarantees, and the losses that (aren't) convex.
Constrained Optimization & KKT
Lagrange multipliers, inequality constraints, KKT conditions, and how duality connects back to the SVM.
Loss Functions
What we actually optimize — regression losses (MSE, MAE, Huber), classification losses (cross-entropy, hinge, focal), the link between maximum likelihood and loss, and how to choose.
Hyperparameter Optimization & AutoML
Tuning the knobs you don't learn by gradient descent — grid, random, and Bayesian search, early-stopping methods like Hyperband/ASHA, and the idea of automating model design.
Optimization for ML Quiz
Test your understanding of gradient descent variants, convexity, constrained optimization, and loss functions.