BatchNorm & Dropout
Two techniques transformed deep learning from an art into an engineering discipline: Batch Normalization and Dropout. Neither is about the network's hypothesis class. They do not change what functions the network can represent. Instead, they change the optimization landscape, making it easier to find good solutions and harder to memorize noise.
The problem: internal covariate shift
Each layer of a deep network receives inputs whose distribution shifts every time the weights of the previous layer change. This internal covariate shift slows training because later layers must constantly adapt to a moving target.
Settled vs folklore
Internal covariate shift is the original (2015) motivation for batch norm, and it's the story you'll hear most often, but it is contested. A well-known 2018 study ("How Does Batch Normalization Help Optimization?") showed you can inject covariate shift after batch norm and still train fine, and argued the real mechanism is a smoother loss landscape — a genuine optimization effect. Treat covariate shift as useful intuition, not settled fact; the smoothing account is where the field currently sits.
Concretely, imagine training a five-layer network. After one gradient step, the first layer's weights change slightly, which changes the distribution of activations entering the second layer, which changes what the second layer needs to do, which changes the gradients the third layer sees — a cascade that makes each layer's job harder than it needs to be.
Symptoms in practice:
- Saturating activations (sigmoid/tanh outputs stuck near ±1, gradient ≈ 0)
- Need for very careful weight initialization
- Need for very small learning rates
- Training instability that worsens with depth
Batch Normalization
Batch Normalization (Ioffe & Szegedy, 2015) normalizes the pre-activations of each layer across the current mini-batch, then re-scales with learnable parameters:
where and are the mini-batch mean and variance, and are learned per-feature scale and shift.
Why two steps? The normalization step forces zero mean and unit variance. But sometimes the network needs a non-zero mean or non-unit variance for its activations — the learned restore that flexibility. At the limit, if and , the layer has learned to undo the normalization entirely, but in practice this never happens because the smooth optimization landscape encourages smaller, useful values.
Inference mode: During training, BatchNorm uses mini-batch statistics, which vary each step. At inference time, the batch may contain just one sample, making batch statistics meaningless. Instead, BatchNorm accumulates running statistics during training: (exponential moving average). At inference, these fixed running statistics replace and , making predictions deterministic. This is why calling model.eval() is essential — it switches the mode.
For the 4-element worked example (, , , output with , ), and a NumPy from-scratch implementation:
Train vs. eval mode
Forgetting to call model.eval() in PyTorch before inference is a common bug. In train mode, BatchNorm uses batch statistics (stochastic). In eval mode, it uses running statistics (deterministic). Failing to switch causes random-looking inference errors, especially on single-sample inputs.
What BatchNorm buys you
- Higher learning rates: the normalized landscape has more consistent curvature
- Less sensitivity to initialization: bad initial weights are corrected at the first forward pass
- Mild regularization: batch statistics add noise (each example is normalized relative to other random examples in the batch)
- Faster convergence: often 5-10× fewer iterations to the same accuracy
import numpy as np
def batchnorm_forward(x, gamma, beta, eps=1e-5):
"""BatchNorm forward pass. x: (N, D) mini-batch."""
mu = x.mean(axis=0)
var = x.var(axis=0)
x_hat = (x - mu) / np.sqrt(var + eps)
return gamma * x_hat + beta
Dropout
Dropout (Srivastava et al., 2014) randomly zeroes a fraction of a layer's activations during each training forward pass — each neuron is independently retained with probability and dropped (set to zero) with probability .
The division by is inverted dropout — it keeps the expected activation magnitude the same, so no scaling is needed at test time. At inference, dropout is disabled and all neurons are used (with the original weights, unscaled).
The ensemble interpretation
Dropout is equivalent to training an ensemble of different networks (where is the number of dropped neurons) with shared weights. Each forward pass samples a different "thinned network" from this exponentially large ensemble.
Why does test-time averaging approximate the ensemble mean? Consider one neuron with weight and input . During training with dropout rate , inverted dropout scales active neurons by :
- With probability : contribution =
- With probability : dropped, contribution =
Expected contribution = .
At test time all neurons are active and unchanged (no scaling needed) — each contributes . The test-time output therefore equals the expected training-time output, which is the expectation over all ensemble members.
Ensembles reduce variance — different ensemble members make different errors, and averaging them out reduces overfitting. Dropout achieves this benefit implicitly, at nearly zero additional cost.
The regularization interpretation
Neurons with dropout cannot rely on any specific other neuron always being present. This forces the network to learn redundant representations — multiple neurons must each independently encode useful features. This is mathematically similar to L1 and L2 regularization:
| Method | Mechanism | Effect |
|---|---|---|
| L2 regularization | Penalizes large weights | Shrinks all weights toward zero |
| L1 regularization | Penalizes weight magnitude | Sparsifies weights |
| Dropout | Randomly zeros activations | Forces redundant, independent features |
In fact, for a linear model, dropout with rate applied to inputs is equivalent to L2 regularization with penalty proportional to .
Typical dropout rates
A dropout rate of (drop 50% of neurons) is standard for fully-connected layers. For convolutional layers, – is more common — spatial features are spatially correlated, so dropping entire feature maps (SpatialDropout) is more effective than dropping individual pixels.
Worked example: one forward pass
Hidden layer activations: . Dropout rate .
Step 1: sample mask . Say we get (neurons 0 and 2 survive).
Step 2 — apply and scale: .
The scale factor compensates for the halved expected sum, so the downstream layer sees the same expected magnitude regardless of .
def dropout(h, p, training=True, seed=None):
"""Apply inverted dropout. p: drop probability."""
if not training:
return h # inference: no dropout
rng = np.random.RandomState(seed)
mask = (rng.rand(*h.shape) > p).astype(float)
return h * mask / (1.0 - p)
For inverted-dropout scaling in full, the regularization math, and MC dropout as uncertainty estimation:
BatchNorm vs. Dropout: when to use which
| BatchNorm | Dropout | |
|---|---|---|
| Primary effect | Stabilizes optimization | Reduces overfitting |
| Used in | CNNs, dense layers | Dense layers, transformers |
| Combined with | Usually comes before activation | Usually comes after activation |
| Not effective for | Very small batches (< 4) | Convolutional feature maps (use SpatialDropout) |
| Modern substitute | LayerNorm (Transformers) | Weight decay / data augmentation |
For CNNs specifically — where batch size is often small (large images, detection/segmentation) but LayerNorm's "pool everything" reduction is too coarse — GroupNorm and InstanceNorm offer a middle ground, each normalizing per-sample so neither needs running statistics at all:
Ordering: BN before or after activation?
The original BatchNorm paper normalizes pre-activations (before the activation function). In practice, "BN after activation" sometimes works better. Transformers mostly use LayerNorm applied before the sublayer (Pre-LN). Try both and pick whichever stabilizes training in your setting.
Exercise
A mini-batch has pre-activations [1, 3, 5, 7] for one feature. After BatchNorm (with γ=1, β=0), what is the normalized value of 7?
Need a hint?
Compute the batch mean and standard deviation first.
Exercise
With inverted dropout (rate p=0.5), a surviving neuron's activation is multiplied by:
Need a hint?
Inverted dropout keeps expected activation magnitude constant.
Exercise
A network achieves 98% training accuracy but only 72% validation accuracy. Which technique is most likely to help?
Need a hint?
Think about what each technique addresses: optimization vs. overfitting.
Common mistakes
- Leaving dropout on at test time: dropout should be disabled during inference (
model.eval()in PyTorch). Leaving it on gives stochastic predictions and typically worse accuracy. - Applying BatchNorm with batch size 1: with a single example, batch variance is undefined. Use GroupNorm or InstanceNorm for single-sample inference or very small batches.
- Using dropout in place of BatchNorm for instability: they address different problems. If training is unstable (exploding/vanishing gradients), add BatchNorm. If training accuracy is much higher than validation accuracy (overfitting), add dropout.
Key takeaways
- BatchNorm normalizes pre-activations per mini-batch, then re-scales with learnable ; it speeds convergence and reduces sensitivity to initialization.
- At inference, BatchNorm uses running statistics (computed during training) — always call
model.eval(). - Dropout randomly zeroes activations during training; at test time all neurons are active (no scaling needed with inverted dropout).
- Dropout ≈ implicit ensemble of networks; also similar to L2 regularization on the feature representation.
Related concepts
- Layers & the Forward Pass: the layers these techniques modify
- Regularization: L1/L2, the explicit regularization family
- Transformer Architecture: uses LayerNorm instead of BatchNorm