Batch Normalization: Forward Pass & Running Stats
Batch Normalization (Ioffe & Szegedy, 2015) normalizes the pre-activations of each layer across the current mini-batch, then re-scales with learnable parameters. This stabilizes the optimization landscape, enables higher learning rates, and reduces sensitivity to weight initialization.
The three steps of the forward pass
For a mini-batch of pre-activations at one feature:
Step 1 — mini-batch statistics:
Step 2 — normalize:
where prevents division by zero. After this step: , .
Step 3 — rescale with learnable and :
After rescaling: , .
Why two steps? Normalizing alone forces mean 0 and variance 1, which might not be optimal for the downstream layer. The learnable and restore full expressivity — at the limit they can undo the normalization entirely, but the smooth optimization landscape encourages useful, near-identity values.
A worked example (, )
Step 1 — statistics:
Step 2: normalize ( for clarity):
Verify: mean ✓, variance ✓
Step 3: rescale with , :
Verify: mean ✓, variance ✓
Running statistics for inference
During training, batch statistics are computed fresh each mini-batch. At inference time (often a single sample with no batch), BatchNorm switches to running averages accumulated during training via an exponential moving average:
where is the momentum parameter. At inference, the layer uses the fixed , instead of batch statistics, making predictions fully deterministic.
Train vs. eval mode
In PyTorch, always call model.eval() before inference. In train mode, BatchNorm uses stochastic batch statistics. In eval mode, it uses the deterministic running statistics. Forgetting this causes unpredictable inference errors, especially on single-sample inputs where batch variance is undefined.
Implementation from scratch in NumPy
import numpy as np
def batchnorm_forward(x, gamma, beta, eps=1e-5):
"""
x: (N, D) — N samples, D features
gamma: (D,) — per-feature scale
beta: (D,) — per-feature shift
Returns: y (N,D), cache for backward pass
"""
mu = x.mean(axis=0) # (D,)
var = x.var(axis=0) # (D,)
x_hat = (x - mu) / np.sqrt(var + eps)
y = gamma * x_hat + beta
return y, (x, x_hat, mu, var, gamma, eps)
# Reproduce the worked example (1 feature, 4 samples)
x = np.array([[2.0], [4.0], [6.0], [8.0]]) # (4, 1)
gamma, beta = np.array([2.0]), np.array([1.0])
y, _ = batchnorm_forward(x, gamma, beta, eps=0)
print("x_hat:", ((x - x.mean()) / x.std()).ravel().round(3))
print("y: ", y.ravel().round(3)) # [-1.683 0.106 1.894 3.683]
print(f"mean(y)={y.mean():.3f} var(y)={y.var():.3f}") # 1.000, 4.000
# Simulate running statistics accumulation over 5 training batches
mu_run, var_run = 0.0, 1.0
alpha = 0.1
for mu_b, var_b in [(5.0, 5.0), (4.8, 4.9), (5.2, 5.1), (5.0, 4.8), (4.9, 5.0)]:
mu_run = (1 - alpha) * mu_run + alpha * mu_b
var_run = (1 - alpha) * var_run + alpha * var_b
print(f"\nRunning mean after 5 batches: {mu_run:.4f}") # 2.0392 — still 59% low
print(f"Running var after 5 batches: {var_run:.4f}") # 2.6208 — heading to 5, slowly
# The EMA starts at (0, 1), so it needs ~44 batches at alpha=0.1 to come within
# 1% of the truth. Evaluate before then and eval mode normalises with badly
# wrong statistics — see the trace below.
Step through it
BatchNorm — the batch in the forward pass, and the two ways it bites
The page's worked example, run. A four-element batch x = (2, 4, 6, 8) gives μ_B = 5.0 and σ²_B = 5.0, so normalising yields x̂ = (-1.342, -0.447, 0.447, 1.342). Note what the batch statistics are computed **across**: the samples, not the features. Every sample's normalised value depends on every *other* sample in the batch, which is the property that everything else on this page follows from.
The page's worked example run exactly (x = 2,4,6,8 → y = −1.683, 0.106, 1.894, 3.683, with mean(y) = β and var(y) = γ² confirmed), then the consequence of normalising across the batch: the same input gets a different output depending on who it was batched with. Two measured payoffs. The page's own EMA example claims the running mean converges toward 5 and reaches 2.0392 after five batches — a 59% error needing 44 batches to close, which is a commoner cause of broken eval-mode output than forgetting model.eval(). And at batch size 1 the variance is not undefined but exactly zero, so x̂ = 0 and the layer returns β for every input — 2, 40 and −1000 all come out the same.
What to notice:
- Frame 2: the layer contains the identity. Set and and the output returns the input exactly (error 0.0). Normalizing therefore cannot cost the network expressivity; it only changes which solutions are easy to reach.
- Frame 3: the layer is not a function of its input alone. The same value comes out anywhere from 1.871 to 3.136 depending purely on which three samples share its batch. During training that noise is a regularizer; at inference it would mean your prediction depends on who else was in the request batch.
- Frame 5: the running statistics are wrong for a long time. After the five batches above against a true mean of 5, a 59% error, and it takes 44 batches to come within 1%. For a fixed input the eval-mode output drifts 11.99 → 2.79 as the EMA warms up. This is a commoner cause of "eval mode is broken" than forgetting
model.eval(). - Frame 6: at the variance is not undefined, it is exactly zero. So and for every input: 2, 40 and −1000 all return 1.0. Not noisy — constant. The batch-size sweep shows is not the endpoint of the small-batch noise trend but a different failure entirely, and it is why transformers use LayerNorm.
Placement in a network
The original paper normalizes pre-activations (before the nonlinearity): Linear → BN → ReLU. In practice "BN after activation" sometimes works better for residual networks. Transformers use LayerNorm instead (normalized over the feature dimension rather than the batch dimension), which works well at batch size 1.
Related concepts
- BatchNorm & Dropout: the full lesson: internal covariate shift, inverted dropout, ensemble interpretation, train/eval mode
- Layers & the Forward Pass: the layers that BatchNorm is inserted between
- Transformer Architecture: uses LayerNorm (normalizes over features, not the batch)