BatchNorm & DropoutOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

BatchNorm & Dropout

Two techniques transformed deep learning from an art into an engineering discipline: Batch Normalization and Dropout. Neither is about the network's hypothesis class. They do not change what functions the network can represent. Instead, they change the optimization landscape, making it easier to find good solutions and harder to memorize noise.

The problem: internal covariate shift

Each layer of a deep network receives inputs whose distribution shifts every time the weights of the previous layer change. This internal covariate shift slows training because later layers must constantly adapt to a moving target.

Settled vs folklore

Internal covariate shift is the original (2015) motivation for batch norm, and it's the story you'll hear most often, but it is contested. A well-known 2018 study ("How Does Batch Normalization Help Optimization?") showed you can inject covariate shift after batch norm and still train fine, and argued the real mechanism is a smoother loss landscape — a genuine optimization effect. Treat covariate shift as useful intuition, not settled fact; the smoothing account is where the field currently sits.

Concretely, imagine training a five-layer network. After one gradient step, the first layer's weights change slightly, which changes the distribution of activations entering the second layer, which changes what the second layer needs to do, which changes the gradients the third layer sees — a cascade that makes each layer's job harder than it needs to be.

Symptoms in practice:

  • Saturating activations (sigmoid/tanh outputs stuck near ±1, gradient ≈ 0)
  • Need for very careful weight initialization
  • Need for very small learning rates
  • Training instability that worsens with depth

Batch Normalization

Batch Normalization (Ioffe & Szegedy, 2015) normalizes the pre-activations of each layer across the current mini-batch, then re-scales with learnable parameters:

x^i=xi−μBσB2+ϵ,yi=γx^i+β\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}, \qquad y_i = \gamma \hat{x}_i + \beta

where μB=1m∑i=1mxi\mu_B = \frac{1}{m}\sum_{i=1}^m x_i and σB2=1m∑i=1m(xi−μB)2\sigma_B^2 = \frac{1}{m}\sum_{i=1}^m (x_i - \mu_B)^2 are the mini-batch mean and variance, and γ,β\gamma, \beta are learned per-feature scale and shift.

Why two steps? The normalization step forces zero mean and unit variance. But sometimes the network needs a non-zero mean or non-unit variance for its activations — the learned γ,β\gamma, \beta restore that flexibility. At the limit, if γ=σB\gamma = \sigma_B and β=μB\beta = \mu_B, the layer has learned to undo the normalization entirely, but in practice this never happens because the smooth optimization landscape encourages smaller, useful values.

Inference mode: During training, BatchNorm uses mini-batch statistics, which vary each step. At inference time, the batch may contain just one sample, making batch statistics meaningless. Instead, BatchNorm accumulates running statistics during training: μrun←0.9 μrun+0.1 μB\mu_\text{run} \leftarrow 0.9\,\mu_\text{run} + 0.1\,\mu_B (exponential moving average). At inference, these fixed running statistics replace μB\mu_B and σB2\sigma^2_B, making predictions deterministic. This is why calling model.eval() is essential — it switches the mode.

For the 4-element worked example (x=(2,4,6,8)\mathbf{x} = (2,4,6,8), μB=5\mu_B = 5, σB2=5\sigma^2_B = 5, output y≈(−1.683,0.106,1.894,3.683)\mathbf{y} \approx (-1.683, 0.106, 1.894, 3.683) with γ=2\gamma=2, β=1\beta=1), and a NumPy from-scratch implementation:

Deep dive
Batch Normalization: Forward Pass & Running Stats
Includes runnable notebook

Train vs. eval mode

Forgetting to call model.eval() in PyTorch before inference is a common bug. In train mode, BatchNorm uses batch statistics (stochastic). In eval mode, it uses running statistics (deterministic). Failing to switch causes random-looking inference errors, especially on single-sample inputs.

What BatchNorm buys you

  • Higher learning rates: the normalized landscape has more consistent curvature
  • Less sensitivity to initialization: bad initial weights are corrected at the first forward pass
  • Mild regularization: batch statistics add noise (each example is normalized relative to other random examples in the batch)
  • Faster convergence: often 5-10× fewer iterations to the same accuracy
Python
import numpy as np

def batchnorm_forward(x, gamma, beta, eps=1e-5):
    """BatchNorm forward pass. x: (N, D) mini-batch."""
    mu  = x.mean(axis=0)
    var = x.var(axis=0)
    x_hat = (x - mu) / np.sqrt(var + eps)
    return gamma * x_hat + beta

Dropout

Dropout (Srivastava et al., 2014) randomly zeroes a fraction pp of a layer's activations during each training forward pass — each neuron is independently retained with probability (1−p)(1-p) and dropped (set to zero) with probability pp.

ri∼Bernoulli(1−p),h~i=ri⋅hi/(1−p)r_i \sim \text{Bernoulli}(1-p), \qquad \tilde{h}_i = r_i \cdot h_i / (1-p)

The division by (1−p)(1-p) is inverted dropout — it keeps the expected activation magnitude the same, so no scaling is needed at test time. At inference, dropout is disabled and all neurons are used (with the original weights, unscaled).

The ensemble interpretation

Dropout is equivalent to training an ensemble of 2N2^N different networks (where NN is the number of dropped neurons) with shared weights. Each forward pass samples a different "thinned network" from this exponentially large ensemble.

Why does test-time averaging approximate the ensemble mean? Consider one neuron with weight ww and input xx. During training with dropout rate pp, inverted dropout scales active neurons by 1/(1−p)1/(1-p):

  • With probability (1−p)(1-p): contribution = w1−p⋅x\frac{w}{1-p} \cdot x
  • With probability pp: dropped, contribution = 00

Expected contribution = (1−p)⋅wx1−p=wx(1-p) \cdot \frac{wx}{1-p} = wx.

At test time all neurons are active and unchanged (no scaling needed) — each contributes wxwx. The test-time output therefore equals the expected training-time output, which is the expectation over all 2N2^N ensemble members.

Ensembles reduce variance — different ensemble members make different errors, and averaging them out reduces overfitting. Dropout achieves this benefit implicitly, at nearly zero additional cost.

The regularization interpretation

Neurons with dropout cannot rely on any specific other neuron always being present. This forces the network to learn redundant representations — multiple neurons must each independently encode useful features. This is mathematically similar to L1 and L2 regularization:

MethodMechanismEffect
L2 regularizationPenalizes large weightsShrinks all weights toward zero
L1 regularizationPenalizes weight magnitudeSparsifies weights
DropoutRandomly zeros activationsForces redundant, independent features

In fact, for a linear model, dropout with rate pp applied to inputs is equivalent to L2 regularization with penalty proportional to p/(1−p)p/(1-p).

Typical dropout rates

A dropout rate of p=0.5p = 0.5 (drop 50% of neurons) is standard for fully-connected layers. For convolutional layers, p=0.1p = 0.1–0.20.2 is more common — spatial features are spatially correlated, so dropping entire feature maps (SpatialDropout) is more effective than dropping individual pixels.

Worked example: one forward pass

Hidden layer activations: h=(2.0, 1.5, 3.0, 0.5)\mathbf{h} = (2.0,\ 1.5,\ 3.0,\ 0.5). Dropout rate p=0.5p = 0.5.

Step 1: sample mask r∼Bernoulli(0.5)4\mathbf{r} \sim \text{Bernoulli}(0.5)^4. Say we get r=(1,0,1,0)\mathbf{r} = (1, 0, 1, 0) (neurons 0 and 2 survive).

Step 2 — apply and scale: h~=r⊙h/(1−p)=(4.0, 0, 6.0, 0)\tilde{\mathbf{h}} = \mathbf{r} \odot \mathbf{h} / (1-p) = (4.0,\ 0,\ 6.0,\ 0).

The scale factor 1/(1−p)=21/(1-p) = 2 compensates for the halved expected sum, so the downstream layer sees the same expected magnitude regardless of pp.

Python
def dropout(h, p, training=True, seed=None):
    """Apply inverted dropout. p: drop probability."""
    if not training:
        return h                          # inference: no dropout
    rng = np.random.RandomState(seed)
    mask = (rng.rand(*h.shape) > p).astype(float)
    return h * mask / (1.0 - p)

For inverted-dropout scaling in full, the regularization math, and MC dropout as uncertainty estimation:

Deep dive
Dropout: Inverted Dropout, Regularization Mechanism, and MC Dropout
Includes runnable notebook

BatchNorm vs. Dropout: when to use which

BatchNormDropout
Primary effectStabilizes optimizationReduces overfitting
Used inCNNs, dense layersDense layers, transformers
Combined withUsually comes before activationUsually comes after activation
Not effective forVery small batches (< 4)Convolutional feature maps (use SpatialDropout)
Modern substituteLayerNorm (Transformers)Weight decay / data augmentation

For CNNs specifically — where batch size is often small (large images, detection/segmentation) but LayerNorm's "pool everything" reduction is too coarse — GroupNorm and InstanceNorm offer a middle ground, each normalizing per-sample so neither needs running statistics at all:

Deep dive
GroupNorm & InstanceNorm: Batch-Independent Normalization
Includes runnable notebook

Ordering: BN before or after activation?

The original BatchNorm paper normalizes pre-activations (before the activation function). In practice, "BN after activation" sometimes works better. Transformers mostly use LayerNorm applied before the sublayer (Pre-LN). Try both and pick whichever stabilizes training in your setting.

Exercise

A mini-batch has pre-activations [1, 3, 5, 7] for one feature. After BatchNorm (with γ=1, β=0), what is the normalized value of 7?

Need a hint?

Compute the batch mean and standard deviation first.

Exercise

With inverted dropout (rate p=0.5), a surviving neuron's activation is multiplied by:

Need a hint?

Inverted dropout keeps expected activation magnitude constant.

Exercise

A network achieves 98% training accuracy but only 72% validation accuracy. Which technique is most likely to help?

Need a hint?

Think about what each technique addresses: optimization vs. overfitting.

Common mistakes

  1. Leaving dropout on at test time: dropout should be disabled during inference (model.eval() in PyTorch). Leaving it on gives stochastic predictions and typically worse accuracy.
  2. Applying BatchNorm with batch size 1: with a single example, batch variance is undefined. Use GroupNorm or InstanceNorm for single-sample inference or very small batches.
  3. Using dropout in place of BatchNorm for instability: they address different problems. If training is unstable (exploding/vanishing gradients), add BatchNorm. If training accuracy is much higher than validation accuracy (overfitting), add dropout.

Key takeaways

  • BatchNorm normalizes pre-activations per mini-batch, then re-scales with learnable γ,β\gamma, \beta; it speeds convergence and reduces sensitivity to initialization.
  • At inference, BatchNorm uses running statistics (computed during training) — always call model.eval().
  • Dropout randomly zeroes activations during training; at test time all neurons are active (no scaling needed with inverted dropout).
  • Dropout ≈ implicit ensemble of 2N2^N networks; also similar to L2 regularization on the feature representation.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
BatchNorm & DropoutLinear Algebra for MLCalculus for MLLayers & the Forward PassRegularization: Ridge & L…The Transformer Architect…Weight Initialization & T…Modern CNN ArchitecturesBatch Normalization: Forw…Dropout: Inverted Dropout…GroupNorm & InstanceNorm:…