Loss Functions
Every optimizer in this course minimizes something, but what? This lesson is the one place the course steps out of the optimization slot and into the objective slot: the loss function turns "the model is wrong" into a single number whose gradient drives learning. The choice of loss is one of the most consequential modeling decisions you make: it encodes what kinds of errors you care about, how you treat outliers, and even what probabilistic assumption you're making about the data.
Real-world examples: MSE in every regression, cross-entropy in essentially every classifier and language model, hinge loss in SVMs, Huber loss in robust regression and RL, focal loss in object detectors with class imbalance.
What a loss must do
A loss measures the discrepancy between a prediction and a target , and the training objective averages it over the data:
For gradient-based learning it should be differentiable (almost everywhere) and, ideally, give informative gradients even when predictions are badly wrong.
Regression losses
| Loss | Formula | Behavior |
|---|---|---|
| MSE (L2) | smooth; penalizes large errors quadratically → outlier-sensitive | |
| MAE (L1) | constant gradient; robust to outliers; non-smooth at 0 | |
| Huber | quadratic for small errors, linear beyond | best of both: smooth near 0, robust to large errors |
The loss decides what 'average' you predict
Minimizing MSE makes the model predict the conditional mean; minimizing MAE makes it predict the conditional median (which is why MAE shrugs off outliers). The loss isn't just a knob — it changes the statistic your model estimates.
Classification losses
For classification we want to penalize confident wrong predictions, which calls for losses over probabilities or margins.
Cross-entropy (log loss) is the default. For a true class with predicted probability , the loss is — it explodes as the model becomes confidently wrong, giving strong gradients exactly when needed. For binary labels:
Hinge loss (SVMs) penalizes predictions that aren't correct by a margin: for . Zero loss once a point is correctly classified beyond the margin — it cares about the boundary, not calibrated probabilities.
Focal loss down-weights easy examples (), so a detector swamped by thousands of easy background boxes still learns from the rare hard positives. It's cross-entropy built for extreme class imbalance.
The deep link: loss = negative log-likelihood
Most losses aren't arbitrary — they're maximum likelihood under an assumed noise model (from the probability course):
- Gaussian noise on the target ⇒ minimizing negative log-likelihood is MSE.
- Laplace noise ⇒ MAE.
- Bernoulli/categorical label ⇒ cross-entropy.
So "choosing a loss" is really "choosing a probabilistic model of your data." The gradient-descent variants from Lesson 1 then minimize whichever surface that choice produces, and the loss shapes that surface:
Gradient descent on f(x) = x²
Each step moves the ball by −learning-rate × gradient. Push the learning rate past 1.0 to watch it diverge.
Choosing a loss
- Regression with outliers? MAE or Huber, not MSE.
- Want calibrated class probabilities? Cross-entropy.
- Want a max-margin boundary? Hinge.
- Severe class imbalance? Weighted cross-entropy or focal loss.
- Add regularization (L2/L1 penalty) to the loss to control overfitting — the loss is also where priors enter (MAP).
Exercise
Your regression targets contain occasional extreme outliers. Why might MAE (or Huber) be a better loss than MSE?
Need a hint?
How does each loss grow with the size of an error?
Exercise
Why is cross-entropy preferred over squared error for training a classifier?
Need a hint?
What gradient does each give when the model is confidently wrong?
Exercise
Minimizing mean squared error is equivalent to maximum likelihood under which assumption about the target noise?
Need a hint?
Which distribution's negative log-likelihood is a sum of squared errors?
Common mistakes
- MSE on classification. Squared error on class labels gives weak gradients when confidently wrong and assumes Gaussian noise on a 0/1 target — use cross-entropy.
- Ignoring outliers with MSE. A few extreme points dominate the quadratic loss and drag the fit; switch to Huber/MAE.
- Class imbalance under plain cross-entropy. The majority class dominates the average loss; reweight or use focal loss.
- Forgetting numerical stability. Compute cross-entropy from logits (log-sum-exp), not by taking of a separately-computed probability that may underflow to 0.
Key takeaways
- The loss function defines the objective the optimizer minimizes; it encodes which errors matter and what data model you assume.
- Regression: MSE (mean, outlier-sensitive), MAE (median, robust), Huber (a smooth compromise).
- Classification: cross-entropy (default, calibrated probabilities), hinge (max-margin), focal (class imbalance).
- Most losses are negative log-likelihood under a noise model (Gaussian→MSE, Laplace→MAE, Bernoulli→cross-entropy), so picking a loss is picking a probabilistic model.
Related concepts
- Gradient Descent Variants: the optimizers that minimize whichever loss you choose
- Maximum Likelihood Estimation: the likelihood view that derives these losses
- Maximum Margin: where hinge loss comes from