Gradient Descent
Training a neural network means finding weights that minimize a loss function — a number that measures how wrong the network's predictions are. Gradient descent is the algorithm that does this, and it is the engine behind essentially all of deep learning.
Real-world examples: Every trained model (image classifiers, language models, recommenders) was fit by some variant of gradient descent.
The core idea
Imagine standing on a hilly landscape in fog. You can feel the slope under your feet. The steepest downhill direction is the negative gradient. Take a small step that way. Repeat until you're in a valley.
Here is the learning rate — how big each step is. We'll see exactly how it controls speed versus stability after the worked example below.
Warm-up: a single-parameter bowl
Loss , so the gradient is . Starting at with :
The negative gradient is positive-pushing here, so moves toward 5, the minimum. Each step shrinks the distance to the optimum. That is the whole mechanic — now let's apply it to a real model.
Worked example: fitting a line with MSE
Almost every "loss" you'll meet is the mean squared error (MSE) of a model's predictions. Let's derive the update rule for the simplest model, a straight line, and then run two iterations entirely by hand.
1. The model and the loss
Model with two parameters, a slope and an intercept :
Over training examples , the MSE loss is the average squared gap between prediction and target:
2. Deriving the gradients
We need and . Apply the chain rule to one term . Let be the residual. The outer derivative of is ; the inner derivative of with respect to is , and with respect to is :
Averaging over all examples gives the full gradients:
The update rule applies the same step to both parameters at once:
3. The data
Three points that lie exactly on the line (so we know the answer is , ):
| 1 | 3 |
| 2 | 5 |
| 3 | 7 |
We start from , with learning rate . At the start the predictions are all , so the residuals are and the loss is
4. Iteration 1, by hand
With the residuals are . Plug into the boxed formulas:
Now step downhill (subtract times each gradient):
The new loss is — a single step took us from down to because the gradients were large.
5. Iteration 2, by hand
Recompute the residuals with :
Step again:
Loss drops to . Notice the gradients shrank from to : as you approach the minimum the slope flattens, so the steps automatically get smaller. Keep iterating and converges to exactly .
# Linear-model MSE gradient descent — reproduces the hand calculation above.
data = [(1, 3), (2, 5), (3, 7)] # points on y = 2x + 1
def gradients(w, b):
n = len(data)
dw = (2 / n) * sum((w * x + b - y) * x for x, y in data)
db = (2 / n) * sum((w * x + b - y) for x, y in data)
return dw, db
def loss(w, b):
n = len(data)
return sum((w * x + b - y) ** 2 for x, y in data) / n
w, b, lr = 0.0, 0.0, 0.1
for step in range(2):
dw, db = gradients(w, b)
w, b = w - lr * dw, b - lr * db
print("step {}: w={:.3f} b={:.3f} L={:.4f}".format(step + 1, w, b, loss(w, b)))
# step 1: w=2.267 b=1.000 L=0.3319
# step 2: w=2.018 b=0.893 L=0.0053
The learning rate, concretely
The learning rate scales every step. Using the same line-fitting problem, here is what 50 steps from produce for different :
| Behavior after 50 steps | Final loss | |
|---|---|---|
| Crawls: barely moved () | ||
| Converges smoothly to | ||
| Overshoots and diverges to NaN |
Learning rate matters
Too large: each step overshoots the minimum and lands further out than it started, so the loss grows every step and explodes to NaN. Too small: training is stable but painfully slow and can stall in shallow minima. When training diverges, the first fix is to lower by 10×.
In practice is rarely held constant. It is scheduled to start large and shrink, often with a warmup ramp first:
Batch, stochastic, and mini-batch
In the example above we summed the gradient over all three points before each step. That is full-batch gradient descent. With millions of examples, summing over the whole dataset for a single step is far too slow, so we estimate the gradient from a subset. The number of examples per step defines the variant:
| Variant | Examples per step | Trade-off |
|---|---|---|
| Batch | All () | Exact gradient, smooth path, slow & memory-heavy |
| Stochastic (SGD) | One | Noisy, cheap, the noise can jump out of shallow minima |
| Mini-batch | 32–512 | Best of both: the practical default |
The key insight: a mini-batch gradient is an unbiased estimate of the full gradient. On average it points the right way; any single step is noisy, but over many steps the noise averages out while each step costs a fraction of the full-batch price. With examples and batch size , one pass over the data (an epoch) does updates instead of just one, so mini-batch SGD makes far more progress per pass.
Beyond vanilla: momentum and Adam
Plain SGD crawls through narrow ravines — regions where the loss is very steep in one direction (say, along ) but nearly flat in another (say, along ). It bounces back and forth across the steep walls while barely advancing along the flat valley floor. Smarter optimizers fix this:
- Momentum keeps a running velocity that accumulates past gradients, so consistent downhill directions build up speed while side-to-side oscillations cancel out: The decay (typically ) sets how much past momentum carries over. One-step example: starting from , , with and : first, , then . Next step with again: — momentum has nearly doubled the effective step size by building on the previous gradient. Think of a ball rolling downhill: it gathers speed on long slopes and isn't deflected by small bumps.
- Adam combines momentum with a per-parameter adaptive learning rate — it divides each parameter's step by a running estimate of that parameter's gradient magnitude, so flat directions get larger effective steps and steep directions get smaller ones. This is why Adam handles ravines without manual tuning.
Which optimizer?
Start with Adam — it works well out of the box and is the default for most deep learning. Well-tuned SGD with momentum can generalize slightly better on some vision tasks, but Adam needs far less tuning.
For the full lineage (Adagrad, RMSprop, Adam's bias correction, and AdamW), with every update rule and when each fails:
To watch the three of them descend the same ravine step by step — velocity building in momentum, Adam's per-parameter denominators pulling the steep direction back — Gradient Descent Variants runs all three side by side in a steppable trace.
See it run
Watch gradient descent move across a 2D loss surface — small learning rates inch along, large ones zig-zag or diverge. The companion notebook plots the loss surface, implements the optimizers, and lets you change the learning rate and re-run.
Gradient descent on f(x) = x²
Each step moves the ball by −learning-rate × gradient. Push the learning rate past 1.0 to watch it diverge.
Run it yourself
Open the gradient descent notebook in Colab. Edit the lr value in the "Effect of learning rate" cell and re-run to see the path change live.
Exercise
What learning rate is generally a good starting point for gradient descent?
Need a hint?
Think about the scale: 0.001, 0.01, 0.1, or 10?
Exercise
Set the momentum coefficient β for SGD with momentum (not Adam).
Need a hint?
β = 0 is standard SGD with no memory. β → 1 gives the optimizer infinitely long memory.
Exercise
If the loss is L(w) = (w - 5)² and the current weight is w = 2, what is the gradient?
Need a hint?
Take the derivative of L with respect to w.
Exercise
Which optimizer adapts the learning rate per parameter and includes momentum?
Need a hint?
It's the most popular optimizer in deep learning.
Common mistakes
- Learning rate too high: the loss diverges to NaN. If that happens, lower by 10×.
- Forgetting to shuffle mini-batches — ordered data biases the gradient and slows convergence.
- Not normalizing inputs: features on wildly different scales distort the loss surface into ravines.
Key takeaways
- Gradient descent updates parameters opposite the gradient: , applied to every parameter ( and ) at once.
- For MSE on a linear model, and — averaged residuals, weighted by the input for .
- Gradients shrink as you near the minimum, so steps automatically slow down even with a fixed .
- The learning rate is the most important hyperparameter — too big diverges to NaN, too small crawls.
- Mini-batch SGD is the practical default; momentum and Adam speed convergence.
- Normalize inputs and tune first when training stalls.
Related concepts
- What is a Neuron?: the unit whose weights we optimize
- Linear Regression: minimizing a squared-error loss in closed form vs by descent
- BPTT & the Vanishing Gradient: what happens when gradients flow through many steps