Gradient DescentOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Gradient Descent

Training a neural network means finding weights that minimize a loss function — a number that measures how wrong the network's predictions are. Gradient descent is the algorithm that does this, and it is the engine behind essentially all of deep learning.

Real-world examples: Every trained model (image classifiers, language models, recommenders) was fit by some variant of gradient descent.

The core idea

Imagine standing on a hilly landscape in fog. You can feel the slope under your feet. The steepest downhill direction is the negative gradient. Take a small step that way. Repeat until you're in a valley.

w←w−η⋅∂L∂ww \leftarrow w - \eta \cdot \frac{\partial L}{\partial w}

Here η\eta is the learning rate — how big each step is. We'll see exactly how it controls speed versus stability after the worked example below.

Warm-up: a single-parameter bowl

Loss L(w)=(w−5)2L(w) = (w - 5)^2, so the gradient is dLdw=2(w−5)\frac{dL}{dw} = 2(w - 5). Starting at w=2w = 2 with η=0.1\eta = 0.1:

∇=2(2−5)=−6⇒w←2−0.1 (−6)=2.6\nabla = 2(2 - 5) = -6 \quad\Rightarrow\quad w \leftarrow 2 - 0.1\,(-6) = 2.6

The negative gradient is positive-pushing here, so ww moves toward 5, the minimum. Each step shrinks the distance to the optimum. That is the whole mechanic — now let's apply it to a real model.

Worked example: fitting a line with MSE

Almost every "loss" you'll meet is the mean squared error (MSE) of a model's predictions. Let's derive the update rule for the simplest model, a straight line, and then run two iterations entirely by hand.

1. The model and the loss

Model with two parameters, a slope ww and an intercept bb:

y^i=w xi+b\hat{y}_i = w\,x_i + b

Over nn training examples (xi,yi)(x_i, y_i), the MSE loss is the average squared gap between prediction and target:

L(w,b)=1n∑i=1n(y^i−yi)2=1n∑i=1n(wxi+b−yi)2L(w, b) = \frac{1}{n}\sum_{i=1}^{n}\bigl(\hat{y}_i - y_i\bigr)^2 = \frac{1}{n}\sum_{i=1}^{n}\bigl(w x_i + b - y_i\bigr)^2

2. Deriving the gradients

We need ∂L∂w\dfrac{\partial L}{\partial w} and ∂L∂b\dfrac{\partial L}{\partial b}. Apply the chain rule to one term (wxi+b−yi)2\bigl(w x_i + b - y_i\bigr)^2. Let ri=wxi+b−yir_i = w x_i + b - y_i be the residual. The outer derivative of ri2r_i^2 is 2ri2 r_i; the inner derivative of rir_i with respect to ww is xix_i, and with respect to bb is 11:

∂∂w(ri2)=2ri⋅xi,∂∂b(ri2)=2ri⋅1\frac{\partial}{\partial w}\bigl(r_i^2\bigr) = 2 r_i \cdot x_i, \qquad \frac{\partial}{\partial b}\bigl(r_i^2\bigr) = 2 r_i \cdot 1

Averaging over all nn examples gives the full gradients:

  ∂L∂w=2n∑i=1n(wxi+b−yi) xi    ∂L∂b=2n∑i=1n(wxi+b−yi)  \boxed{\;\frac{\partial L}{\partial w} = \frac{2}{n}\sum_{i=1}^{n}\bigl(w x_i + b - y_i\bigr)\,x_i\;} \qquad \boxed{\;\frac{\partial L}{\partial b} = \frac{2}{n}\sum_{i=1}^{n}\bigl(w x_i + b - y_i\bigr)\;}

The update rule applies the same step to both parameters at once:

w←w−η ∂L∂w,b←b−η ∂L∂bw \leftarrow w - \eta\,\frac{\partial L}{\partial w}, \qquad b \leftarrow b - \eta\,\frac{\partial L}{\partial b}

3. The data

Three points that lie exactly on the line y=2x+1y = 2x + 1 (so we know the answer is w=2w = 2, b=1b = 1):

xix_iyiy_i
13
25
37

We start from w=0w = 0, b=0b = 0 with learning rate η=0.1\eta = 0.1. At the start the predictions are all 00, so the residuals are r=(0−3, 0−5, 0−7)=(−3,−5,−7)r = (0-3,\,0-5,\,0-7) = (-3, -5, -7) and the loss is

L=(−3)2+(−5)2+(−7)23=9+25+493=833≈27.67L = \frac{(-3)^2 + (-5)^2 + (-7)^2}{3} = \frac{9 + 25 + 49}{3} = \frac{83}{3} \approx 27.67

4. Iteration 1, by hand

With w=0,b=0w = 0, b = 0 the residuals are ri=−yir_i = -y_i. Plug into the boxed formulas:

∂L∂w=23[(−3)(1)+(−5)(2)+(−7)(3)]=23(−3−10−21)=23(−34)=−683≈−22.67\frac{\partial L}{\partial w} = \frac{2}{3}\Bigl[(-3)(1) + (-5)(2) + (-7)(3)\Bigr] = \frac{2}{3}(-3 - 10 - 21) = \frac{2}{3}(-34) = -\frac{68}{3} \approx -22.67 ∂L∂b=23[(−3)+(−5)+(−7)]=23(−15)=−10\frac{\partial L}{\partial b} = \frac{2}{3}\Bigl[(-3) + (-5) + (-7)\Bigr] = \frac{2}{3}(-15) = -10

Now step downhill (subtract η\eta times each gradient):

w←0−0.1 (−22.67)=2.267,b←0−0.1 (−10)=1.0w \leftarrow 0 - 0.1\,(-22.67) = 2.267, \qquad b \leftarrow 0 - 0.1\,(-10) = 1.0

The new loss is L≈0.332L \approx 0.332 — a single step took us from 27.6727.67 down to 0.330.33 because the gradients were large.

5. Iteration 2, by hand

Recompute the residuals with w=2.267,b=1.0w = 2.267, b = 1.0:

r1=2.267(1)+1−3=0.267r2=2.267(2)+1−5=0.533r3=2.267(3)+1−7=0.800\begin{aligned} r_1 &= 2.267(1) + 1 - 3 = 0.267 \\ r_2 &= 2.267(2) + 1 - 5 = 0.533 \\ r_3 &= 2.267(3) + 1 - 7 = 0.800 \end{aligned} ∂L∂w=23[0.267(1)+0.533(2)+0.800(3)]=23(3.733)≈2.489\frac{\partial L}{\partial w} = \frac{2}{3}\bigl[0.267(1) + 0.533(2) + 0.800(3)\bigr] = \frac{2}{3}(3.733) \approx 2.489 ∂L∂b=23[0.267+0.533+0.800]=23(1.6)≈1.067\frac{\partial L}{\partial b} = \frac{2}{3}\bigl[0.267 + 0.533 + 0.800\bigr] = \frac{2}{3}(1.6) \approx 1.067

Step again:

w←2.267−0.1 (2.489)=2.018,b←1.0−0.1 (1.067)=0.893w \leftarrow 2.267 - 0.1\,(2.489) = 2.018, \qquad b \leftarrow 1.0 - 0.1\,(1.067) = 0.893

Loss drops to L≈0.0053L \approx 0.0053. Notice the gradients shrank from ∼23\sim 23 to ∼2.5\sim 2.5: as you approach the minimum the slope flattens, so the steps automatically get smaller. Keep iterating and (w,b)(w, b) converges to exactly (2,1)(2, 1).

Python
# Linear-model MSE gradient descent — reproduces the hand calculation above.
data = [(1, 3), (2, 5), (3, 7)]   # points on y = 2x + 1

def gradients(w, b):
    n = len(data)
    dw = (2 / n) * sum((w * x + b - y) * x for x, y in data)
    db = (2 / n) * sum((w * x + b - y)     for x, y in data)
    return dw, db

def loss(w, b):
    n = len(data)
    return sum((w * x + b - y) ** 2 for x, y in data) / n

w, b, lr = 0.0, 0.0, 0.1
for step in range(2):
    dw, db = gradients(w, b)
    w, b = w - lr * dw, b - lr * db
    print("step {}: w={:.3f} b={:.3f} L={:.4f}".format(step + 1, w, b, loss(w, b)))
# step 1: w=2.267 b=1.000 L=0.3319
# step 2: w=2.018 b=0.893 L=0.0053

The learning rate, concretely

The learning rate η\eta scales every step. Using the same line-fitting problem, here is what 50 steps from (w,b)=(0,0)(w,b)=(0,0) produce for different η\eta:

η\etaBehavior after 50 stepsFinal loss
0.0010.001Crawls: barely moved (w≈0.87w \approx 0.87)≈9.07\approx 9.07
0.10.1Converges smoothly to (2,1)(2, 1)≈0.0001\approx 0.0001
0.50.5Overshoots and diverges to NaN∞\infty

Learning rate matters

Too large: each step overshoots the minimum and lands further out than it started, so the loss grows every step and explodes to NaN. Too small: training is stable but painfully slow and can stall in shallow minima. When training diverges, the first fix is to lower η\eta by 10×.

In practice η\eta is rarely held constant. It is scheduled to start large and shrink, often with a warmup ramp first:

Deep dive
Learning Rate Schedules: Warmup, Decay & Restarts
Includes runnable notebook

Batch, stochastic, and mini-batch

In the example above we summed the gradient over all three points before each step. That is full-batch gradient descent. With millions of examples, summing over the whole dataset for a single step is far too slow, so we estimate the gradient from a subset. The number of examples per step defines the variant:

VariantExamples per stepTrade-off
BatchAll (nn)Exact gradient, smooth path, slow & memory-heavy
Stochastic (SGD)OneNoisy, cheap, the noise can jump out of shallow minima
Mini-batch32–512Best of both: the practical default

The key insight: a mini-batch gradient is an unbiased estimate of the full gradient. On average it points the right way; any single step is noisy, but over many steps the noise averages out while each step costs a fraction of the full-batch price. With nn examples and batch size BB, one pass over the data (an epoch) does n/Bn/B updates instead of just one, so mini-batch SGD makes far more progress per pass.

Beyond vanilla: momentum and Adam

Plain SGD crawls through narrow ravines — regions where the loss is very steep in one direction (say, along w1w_1) but nearly flat in another (say, along w2w_2). It bounces back and forth across the steep walls while barely advancing along the flat valley floor. Smarter optimizers fix this:

  • Momentum keeps a running velocity vv that accumulates past gradients, so consistent downhill directions build up speed while side-to-side oscillations cancel out: v←βv−η ∇L,w←w+vv \leftarrow \beta v - \eta\,\nabla L, \qquad w \leftarrow w + v The decay β\beta (typically 0.90.9) sets how much past momentum carries over. One-step example: starting from w=0w=0, v=0v=0, with η=0.1\eta=0.1 and ∇L=−2\nabla L = -2: first, v←0.9(0)−0.1(−2)=0.2v \leftarrow 0.9(0) - 0.1(-2) = 0.2, then w←0+0.2=0.2w \leftarrow 0 + 0.2 = 0.2. Next step with ∇L=−2\nabla L = -2 again: v←0.9(0.2)+0.2=0.38v \leftarrow 0.9(0.2) + 0.2 = 0.38 — momentum has nearly doubled the effective step size by building on the previous gradient. Think of a ball rolling downhill: it gathers speed on long slopes and isn't deflected by small bumps.
  • Adam combines momentum with a per-parameter adaptive learning rate — it divides each parameter's step by a running estimate of that parameter's gradient magnitude, so flat directions get larger effective steps and steep directions get smaller ones. This is why Adam handles ravines without manual tuning.

Which optimizer?

Start with Adam — it works well out of the box and is the default for most deep learning. Well-tuned SGD with momentum can generalize slightly better on some vision tasks, but Adam needs far less tuning.

For the full lineage (Adagrad, RMSprop, Adam's bias correction, and AdamW), with every update rule and when each fails:

Deep dive
Optimizers: SGD, Momentum, Adagrad, RMSprop & Adam
Includes runnable notebook

To watch the three of them descend the same ravine step by step — velocity building in momentum, Adam's per-parameter denominators pulling the steep direction back — Gradient Descent Variants runs all three side by side in a steppable trace.

See it run

Watch gradient descent move across a 2D loss surface — small learning rates inch along, large ones zig-zag or diverge. The companion notebook plots the loss surface, implements the optimizers, and lets you change the learning rate and re-run.

Gradient descent on f(x) = x²

weight xloss
step0
x-4.200
loss17.640

Each step moves the ball by −learning-rate × gradient. Push the learning rate past 1.0 to watch it diverge.

Run it yourself

Open the gradient descent notebook in Colab. Edit the lr value in the "Effect of learning rate" cell and re-run to see the path change live.

Exercise

What learning rate is generally a good starting point for gradient descent?

0.00010.501
Need a hint?

Think about the scale: 0.001, 0.01, 0.1, or 10?

Exercise

Set the momentum coefficient β for SGD with momentum (not Adam).

00.501
Need a hint?

β = 0 is standard SGD with no memory. β → 1 gives the optimizer infinitely long memory.

Exercise

If the loss is L(w) = (w - 5)² and the current weight is w = 2, what is the gradient?

Need a hint?

Take the derivative of L with respect to w.

Exercise

Which optimizer adapts the learning rate per parameter and includes momentum?

Need a hint?

It's the most popular optimizer in deep learning.

Common mistakes

  1. Learning rate too high: the loss diverges to NaN. If that happens, lower η\eta by 10×.
  2. Forgetting to shuffle mini-batches — ordered data biases the gradient and slows convergence.
  3. Not normalizing inputs: features on wildly different scales distort the loss surface into ravines.

Key takeaways

  • Gradient descent updates parameters opposite the gradient: w←w−η ∇Lw \leftarrow w - \eta\,\nabla L, applied to every parameter (ww and bb) at once.
  • For MSE on a linear model, ∂L∂w=2n∑(wxi+b−yi) xi\frac{\partial L}{\partial w} = \frac{2}{n}\sum (w x_i + b - y_i)\,x_i and ∂L∂b=2n∑(wxi+b−yi)\frac{\partial L}{\partial b} = \frac{2}{n}\sum (w x_i + b - y_i) — averaged residuals, weighted by the input for ww.
  • Gradients shrink as you near the minimum, so steps automatically slow down even with a fixed η\eta.
  • The learning rate is the most important hyperparameter — too big diverges to NaN, too small crawls.
  • Mini-batch SGD is the practical default; momentum and Adam speed convergence.
  • Normalize inputs and tune η\eta first when training stalls.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Gradient DescentLinear Algebra for MLCalculus for MLDerivatives and GradientsChain Rule and Backpropag…Multivariable OptimizationMaximum Likelihood Estima…What is a Neuron?Gradient Descent VariantsLearning Rate Schedules: …Gradient Descent Optimize…Newton's Method for Optim…OLS: The Normal Equation