Optimization for ML Quiz
Check your understanding of the optimizer zoo — what Adam's bias correction fixes, when convexity guarantees a global minimum, and what KKT conditions actually say.
Exercise
Adam uses β₁=0.9, β₂=0.999, ε=1e-8 by default. What is the role of ε in the update θ_t = θ_{t-1} − η·m̂_t/(√v̂_t + ε)?
Need a hint?
What happens when v̂_t ≈ 0 (parameter never received a significant gradient)?
Exercise
Is the cross-entropy loss L(w) = -log σ(yᵀw) convex in the weight vector w (for binary logistic regression)?
Need a hint?
Compute the Hessian of L with respect to w and check its sign.
Exercise
Which of the following is NOT one of the KKT necessary conditions for a constrained optimization problem?
Need a hint?
The KKT conditions include stationarity, primal feasibility, dual feasibility, and complementary slackness.
Exercise
Cosine annealing decays the learning rate as η_t = η_min + ½(η_max - η_min)(1 + cos(πt/T)). What is the learning rate at t = T (the end of training)?
Need a hint?
Compute cos(π·T/T) = cos(π).
Exercise
At a saddle point of a loss function, what can you say about the eigenvalues of the Hessian?
Need a hint?
A saddle point is neither a local min nor a local max — it goes down in some directions, up in others.
Exercise
You have a budget of 20 training runs to tune 5 hyperparameters, and (unknown to you) only one of them really matters. Why does random search beat grid search here?
Need a hint?
Count the distinct values each method tries along the important dimension.
Exercise
Switching a regression model's loss from MSE to Huber loss (to be robust to outliers) changes which slot of the ML project loop?
Need a hint?
You are redefining what 'good' means, not the model or the search.
Exercise
Transfer test — a technique this course did NOT teach: data augmentation (random crops/flips of training images). Which slot does it modify?
Need a hint?
It expands the training distribution the model sees.