Batch Normalization: Forward Pass & Running StatsOpen in Colab

Batch Normalization: Forward Pass & Running Stats

Batch Normalization (Ioffe & Szegedy, 2015) normalizes the pre-activations of each layer across the current mini-batch, then re-scales with learnable parameters. This stabilizes the optimization landscape, enables higher learning rates, and reduces sensitivity to weight initialization.

The three steps of the forward pass

For a mini-batch of mm pre-activations x1,…,xmx_1, \ldots, x_m at one feature:

Step 1 — mini-batch statistics:

μB=1m∑i=1mxi,σB2=1m∑i=1m(xi−μB)2\mu_B = \frac{1}{m}\sum_{i=1}^m x_i, \qquad \sigma_B^2 = \frac{1}{m}\sum_{i=1}^m (x_i - \mu_B)^2

Step 2 — normalize:

x^i=xi−μBσB2+ϵ\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}

where ϵ≈10−5\epsilon \approx 10^{-5} prevents division by zero. After this step: E[x^]=0\mathbb{E}[\hat{x}] = 0, Var[x^]=1\text{Var}[\hat{x}] = 1.

Step 3 — rescale with learnable γ\gamma and β\beta:

yi=γx^i+βy_i = \gamma \hat{x}_i + \beta

After rescaling: E[y]=β\mathbb{E}[y] = \beta, Var[y]=γ2\text{Var}[y] = \gamma^2.

Why two steps? Normalizing alone forces mean 0 and variance 1, which might not be optimal for the downstream layer. The learnable γ\gamma and β\beta restore full expressivity — at the limit they can undo the normalization entirely, but the smooth optimization landscape encourages useful, near-identity values.

A worked example (x=(2,4,6,8)\mathbf{x} = (2, 4, 6, 8), m=4m = 4)

Step 1 — statistics:

μB=2+4+6+84=5,σB2=(−3)2+(−1)2+12+324=9+1+1+94=5\mu_B = \frac{2+4+6+8}{4} = 5, \qquad \sigma_B^2 = \frac{(-3)^2+(-1)^2+1^2+3^2}{4} = \frac{9+1+1+9}{4} = 5

Step 2: normalize (ϵ=0\epsilon = 0 for clarity):

x^=(2−55, 4−55, 6−55, 8−55)≈(−1.342, −0.447, 0.447, 1.342)\hat{\mathbf{x}} = \left(\frac{2-5}{\sqrt{5}},\ \frac{4-5}{\sqrt{5}},\ \frac{6-5}{\sqrt{5}},\ \frac{8-5}{\sqrt{5}}\right) \approx (-1.342,\ -0.447,\ 0.447,\ 1.342)

Verify: mean =0= 0 ✓, variance =1.3422+0.4472+0.4472+1.34224=1= \frac{1.342^2+0.447^2+0.447^2+1.342^2}{4} = 1 ✓

Step 3: rescale with γ=2\gamma = 2, β=1\beta = 1:

y=2 x^+1≈(−1.683, 0.106, 1.894, 3.683)\mathbf{y} = 2\,\hat{\mathbf{x}} + 1 \approx (-1.683,\ 0.106,\ 1.894,\ 3.683)

Verify: mean =(−1.683+0.106+1.894+3.683)/4=1=β= (−1.683+0.106+1.894+3.683)/4 = 1 = \beta ✓, variance =4=γ2= 4 = \gamma^2 ✓

Running statistics for inference

During training, batch statistics are computed fresh each mini-batch. At inference time (often a single sample with no batch), BatchNorm switches to running averages accumulated during training via an exponential moving average:

μrun←(1−α) μrun+α μB\mu_{\text{run}} \leftarrow (1 - \alpha)\,\mu_{\text{run}} + \alpha\,\mu_B σrun2←(1−α) σrun2+α σB2\sigma^2_{\text{run}} \leftarrow (1 - \alpha)\,\sigma^2_{\text{run}} + \alpha\,\sigma_B^2

where α≈0.1\alpha \approx 0.1 is the momentum parameter. At inference, the layer uses the fixed μrun\mu_{\text{run}}, σrun2\sigma^2_{\text{run}} instead of batch statistics, making predictions fully deterministic.

Train vs. eval mode

In PyTorch, always call model.eval() before inference. In train mode, BatchNorm uses stochastic batch statistics. In eval mode, it uses the deterministic running statistics. Forgetting this causes unpredictable inference errors, especially on single-sample inputs where batch variance is undefined.

Implementation from scratch in NumPy

Python
import numpy as np

def batchnorm_forward(x, gamma, beta, eps=1e-5):
    """
    x:     (N, D) — N samples, D features
    gamma: (D,)   — per-feature scale
    beta:  (D,)   — per-feature shift
    Returns: y (N,D), cache for backward pass
    """
    mu    = x.mean(axis=0)              # (D,)
    var   = x.var(axis=0)               # (D,)
    x_hat = (x - mu) / np.sqrt(var + eps)
    y     = gamma * x_hat + beta
    return y, (x, x_hat, mu, var, gamma, eps)

# Reproduce the worked example (1 feature, 4 samples)
x = np.array([[2.0], [4.0], [6.0], [8.0]])  # (4, 1)
gamma, beta = np.array([2.0]), np.array([1.0])

y, _ = batchnorm_forward(x, gamma, beta, eps=0)
print("x_hat:", ((x - x.mean()) / x.std()).ravel().round(3))
print("y:    ", y.ravel().round(3))   # [-1.683  0.106  1.894  3.683]
print(f"mean(y)={y.mean():.3f}  var(y)={y.var():.3f}")  # 1.000, 4.000

# Simulate running statistics accumulation over 5 training batches
mu_run, var_run = 0.0, 1.0
alpha = 0.1
for mu_b, var_b in [(5.0, 5.0), (4.8, 4.9), (5.2, 5.1), (5.0, 4.8), (4.9, 5.0)]:
    mu_run  = (1 - alpha) * mu_run  + alpha * mu_b
    var_run = (1 - alpha) * var_run + alpha * var_b

print(f"\nRunning mean after 5 batches: {mu_run:.4f}")  # 2.0392 — still 59% low
print(f"Running var  after 5 batches: {var_run:.4f}")  # 2.6208 — heading to 5, slowly

# The EMA starts at (0, 1), so it needs ~44 batches at alpha=0.1 to come within
# 1% of the truth. Evaluate before then and eval mode normalises with badly
# wrong statistics — see the trace below.

Step through it

BatchNorm — the batch in the forward pass, and the two ways it bites

1/6

The page's worked example, run. A four-element batch x = (2, 4, 6, 8) gives μ_B = 5.0 and σ²_B = 5.0, so normalising yields x̂ = (-1.342, -0.447, 0.447, 1.342). Note what the batch statistics are computed **across**: the samples, not the features. Every sample's normalised value depends on every *other* sample in the batch, which is the property that everything else on this page follows from.

python
1# training: statistics from THIS batch
2mu = x.mean(0)
3var = x.var(0)
4xhat = (x - mu) / sqrt(var + eps)
5y = gamma * xhat + beta
6
7# accumulate an estimate for later
8mu_run = (1 - a) * mu_run + a * mu
9var_run = (1 - a) * var_run + a * var
10
11# inference: frozen stats, no batch
12xhat = (x - mu_run) / sqrt(var_run + eps)
13y = gamma * xhat + beta
state
x → x̂
2-1.34
4-0.45
60.45
81.34
batch statistics
m → 4μ_B → 5.0σ²_B → 5.0mean(x̂) → 0.000var(x̂) → 1.000

The page's worked example run exactly (x = 2,4,6,8 → y = −1.683, 0.106, 1.894, 3.683, with mean(y) = β and var(y) = γ² confirmed), then the consequence of normalising across the batch: the same input gets a different output depending on who it was batched with. Two measured payoffs. The page's own EMA example claims the running mean converges toward 5 and reaches 2.0392 after five batches — a 59% error needing 44 batches to close, which is a commoner cause of broken eval-mode output than forgetting model.eval(). And at batch size 1 the variance is not undefined but exactly zero, so x̂ = 0 and the layer returns β for every input — 2, 40 and −1000 all come out the same.

What to notice:

  • Frame 2: the layer contains the identity. Set γ=σB2\gamma = \sqrt{\sigma^2_B} and β=μB\beta = \mu_B and the output returns the input exactly (error 0.0). Normalizing therefore cannot cost the network expressivity; it only changes which solutions are easy to reach.
  • Frame 3: the layer is not a function of its input alone. The same value x=6x = 6 comes out anywhere from 1.871 to 3.136 depending purely on which three samples share its batch. During training that noise is a regularizer; at inference it would mean your prediction depends on who else was in the request batch.
  • Frame 5: the running statistics are wrong for a long time. After the five batches above μrun=2.0392\mu_{\text{run}} = 2.0392 against a true mean of 5, a 59% error, and it takes 44 batches to come within 1%. For a fixed input the eval-mode output drifts 11.99 → 2.79 as the EMA warms up. This is a commoner cause of "eval mode is broken" than forgetting model.eval().
  • Frame 6: at m=1m = 1 the variance is not undefined, it is exactly zero. So x^=0/ϵ=0\hat{x} = 0/\sqrt{\epsilon} = 0 and y=βy = \beta for every input: 2, 40 and −1000 all return 1.0. Not noisy — constant. The batch-size sweep shows m=1m = 1 is not the endpoint of the small-batch noise trend but a different failure entirely, and it is why transformers use LayerNorm.

Placement in a network

The original paper normalizes pre-activations (before the nonlinearity): Linear → BN → ReLU. In practice "BN after activation" sometimes works better for residual networks. Transformers use LayerNorm instead (normalized over the feature dimension rather than the batch dimension), which works well at batch size 1.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Batch Normalization: Forward Pass…Dropout: Inverted Dropout…GroupNorm & InstanceNorm:…BatchNorm & DropoutLayers & the Forward PassThe Transformer Architect…