Optimization for ML Quiz

Optimization for ML Quiz

Check your understanding of the optimizer zoo — what Adam's bias correction fixes, when convexity guarantees a global minimum, and what KKT conditions actually say.

Exercise

Adam uses β₁=0.9, β₂=0.999, ε=1e-8 by default. What is the role of ε in the update θ_t = θ_{t-1} − η·m̂_t/(√v̂_t + ε)?

Need a hint?

What happens when v̂_t ≈ 0 (parameter never received a significant gradient)?

Exercise

Is the cross-entropy loss L(w) = -log σ(yᵀw) convex in the weight vector w (for binary logistic regression)?

Need a hint?

Compute the Hessian of L with respect to w and check its sign.

Exercise

Which of the following is NOT one of the KKT necessary conditions for a constrained optimization problem?

Need a hint?

The KKT conditions include stationarity, primal feasibility, dual feasibility, and complementary slackness.

Exercise

Cosine annealing decays the learning rate as η_t = η_min + ½(η_max - η_min)(1 + cos(πt/T)). What is the learning rate at t = T (the end of training)?

Need a hint?

Compute cos(π·T/T) = cos(π).

Exercise

At a saddle point of a loss function, what can you say about the eigenvalues of the Hessian?

Need a hint?

A saddle point is neither a local min nor a local max — it goes down in some directions, up in others.

Exercise

You have a budget of 20 training runs to tune 5 hyperparameters, and (unknown to you) only one of them really matters. Why does random search beat grid search here?

Need a hint?

Count the distinct values each method tries along the important dimension.

Exercise

Switching a regression model's loss from MSE to Huber loss (to be robust to outliers) changes which slot of the ML project loop?

Need a hint?

You are redefining what 'good' means, not the model or the search.

Exercise

Transfer test — a technique this course did NOT teach: data augmentation (random crops/flips of training images). Which slot does it modify?

Need a hint?

It expands the training distribution the model sees.