GroupNorm & InstanceNorm: Batch-Independent NormalizationOpen in Colab

GroupNorm & InstanceNorm: Batch-Independent Normalization

BatchNorm normalizes over the batch dimension, which breaks down at small batch sizes and needs running statistics for inference. LayerNorm fixes the batch-dependence by normalizing over the feature dimension of each sample independently, but for image tensors, "feature dimension" means all channels and spatial positions at once, which can be too coarse. GroupNorm and InstanceNorm sit between these two: both normalize per sample (so they need no running statistics and work at batch size 1), but they differ in how much of the channel dimension they pool together.

The shared formula

Every member of this normalization family (BatchNorm, LayerNorm, GroupNorm, InstanceNorm) computes the exact same two-step transform. Only the set of elements averaged together, SS, changes:

μS=1∣S∣∑x∈Sx,σS2=1∣S∣∑x∈S(x−μS)2\mu_S = \frac{1}{|S|}\sum_{x \in S} x, \qquad \sigma_S^2 = \frac{1}{|S|}\sum_{x \in S} (x - \mu_S)^2 x^=x−μSσS2+ϵ,y=γ x^+β\hat{x} = \frac{x - \mu_S}{\sqrt{\sigma_S^2 + \epsilon}}, \qquad y = \gamma\,\hat{x} + \beta

with learnable per-channel γ,β\gamma, \beta exactly as in BatchNorm. For a feature map tensor of shape (N,C,H,W)(N, C, H, W) (batch, channels, height, width), the four variants pick SS along different axes:

MethodReduction set SS (what's averaged)Kept independent
BatchNormN,H,WN, H, W: all samples, all spatial positionsone stat per channel cc
LayerNormC,H,WC, H, W: all channels, all spatial positionsone stat per sample nn
GroupNormC/G,H,WC/G, H, W: channels within one group, all spatial positionsone stat per sample nn and group gg
InstanceNormH,WH, W: spatial positions onlyone stat per sample nn and channel cc

GroupNorm: normalize within channel groups, per sample

GroupNorm (Wu & He, 2018) splits the CC channels into GG groups of C/GC/G channels each, then computes one mean/variance per (sample, group) pair, pooling over that group's channels and the spatial dimensions:

μn,g=1(C/G) H W∑c ∈ group g ∑h=1H∑w=1Wxn,c,h,w\mu_{n,g} = \frac{1}{(C/G)\,H\,W} \sum_{c \,\in\, \text{group } g} \ \sum_{h=1}^{H}\sum_{w=1}^{W} x_{n,c,h,w} σn,g2=1(C/G) H W∑c ∈ group g ∑h=1H∑w=1W(xn,c,h,w−μn,g)2\sigma_{n,g}^2 = \frac{1}{(C/G)\,H\,W} \sum_{c \,\in\, \text{group } g} \ \sum_{h=1}^{H}\sum_{w=1}^{W} (x_{n,c,h,w} - \mu_{n,g})^2

Every element in group gg of sample nn is normalized with the same μn,g,σn,g2\mu_{n,g}, \sigma_{n,g}^2, then rescaled with its own channel's γc,βc\gamma_c, \beta_c.

Because the statistics are computed per-sample, GroupNorm's output for one image never depends on any other image in the batch — batch size 1 works identically to batch size 256. This is exactly why it replaces BatchNorm in object detection and segmentation backbones (Mask R-CNN, etc.): high-resolution inputs mean only 1-2 images fit per GPU, too few for BatchNorm's batch statistics to be meaningful.

GroupNorm has two special cases at the extremes of GG:

  • G=1G = 1 (one group containing all channels) — every channel pools together, which is exactly LayerNorm's reduction over (C,H,W)(C, H, W).
  • G=CG = C (each channel is its own group of size 1) — no cross-channel pooling at all, which is exactly InstanceNorm below.

Choosing G

The GroupNorm paper finds G=32G = 32 works well across a wide range of CC (e.g. ResNet-50's 256/512/1024/2048-channel stages), as long as C/GC/G stays reasonably large — too many tiny groups makes the per-group statistics noisy, just like BatchNorm gets noisy at tiny batch sizes.

InstanceNorm: normalize each channel, per sample, alone

InstanceNorm (Ulyanov et al., 2016, originally for style transfer) is the G=CG=C endpoint of GroupNorm: each channel of each sample is normalized using only its own spatial positions — no pooling across channels or across the batch at all:

μn,c=1HW∑h=1H∑w=1Wxn,c,h,w,σn,c2=1HW∑h=1H∑w=1W(xn,c,h,w−μn,c)2\mu_{n,c} = \frac{1}{HW}\sum_{h=1}^{H}\sum_{w=1}^{W} x_{n,c,h,w}, \qquad \sigma_{n,c}^2 = \frac{1}{HW}\sum_{h=1}^{H}\sum_{w=1}^{W} (x_{n,c,h,w} - \mu_{n,c})^2 x^n,c,h,w=xn,c,h,w−μn,cσn,c2+ϵ\hat{x}_{n,c,h,w} = \frac{x_{n,c,h,w} - \mu_{n,c}}{\sqrt{\sigma_{n,c}^2 + \epsilon}}

Why this matters for style transfer and GANs: a feature channel's mean and variance across its own spatial map largely encode contrast and overall intensity — properties of "how this particular image happened to be lit or scaled," not the image's content. Subtracting μn,c\mu_{n,c} and dividing by σn,c\sigma_{n,c} strips that per-instance contrast out, channel by channel, leaving a representation that is invariant to it. Style-transfer methods (and the AdaIN mechanism used in many GAN generators) exploit this directly: they normalize away the content image's own contrast with InstanceNorm, then re-inject a target style's mean and variance via γ,β\gamma, \beta instead of learned constants — swapping one image's "contrast signature" for another's.

InstanceNorm as contrast removal

Scaling and shifting a single channel of a single sample (xn,c,h,w↦a⋅xn,c,h,w+bx_{n,c,h,w} \mapsto a\cdot x_{n,c,h,w} + b for any a>0a > 0) leaves InstanceNorm's output for that channel completely unchanged, because μn,c\mu_{n,c} and σn,c\sigma_{n,c} absorb exactly that transform. That invariance is the formal version of "removes instance-specific contrast," and the notebook below verifies it numerically.

Worked trace

Take a single sample (N=1N=1) with C=4C=4 channels and H=1,W=2H{=}1, W{=}2 spatial positions (2 numbers per channel), grouped into G=2G=2 groups of 2 channels each:

channelvaluesgroup
c0c_0[2, 4][2,\ 4]group 0
c1c_1[6, 10][6,\ 10]group 0
c2c_2[1, 3][1,\ 3]group 1
c3c_3[0, 8][0,\ 8]group 1

InstanceNorm (G=C=4G=C=4, each channel normalized alone over its 2 spatial values):

μ0,0=3, σ2=1⇒c^0=[−1, 1]μ0,1=8, σ2=4⇒c^1=[−1, 1]\mu_{0,0}=3,\ \sigma^2=1 \Rightarrow \hat{c}_0 = [-1,\ 1] \qquad \mu_{0,1}=8,\ \sigma^2=4 \Rightarrow \hat{c}_1 = [-1,\ 1] μ0,2=2, σ2=1⇒c^2=[−1, 1]μ0,3=4, σ2=16⇒c^3=[−1, 1]\mu_{0,2}=2,\ \sigma^2=1 \Rightarrow \hat{c}_2 = [-1,\ 1] \qquad \mu_{0,3}=4,\ \sigma^2=16 \Rightarrow \hat{c}_3 = [-1,\ 1]

Every channel comes out as [−1,1][-1, 1] despite wildly different raw scales (variance 1 vs. variance 16) — exactly the contrast-removal property above: any 2-element vector normalizes to [−1,1][-1,1] regardless of its original mean or spread.

GroupNorm (G=2G=2: group 0 pools c0,c1c_0,c_1; group 1 pools c2,c3c_2,c_3, over both channels and spatial positions at once):

group 0=[2,4,6,10],  μ0,0=5.5, σ0,02=8.75⇒x^≈[−1.183, −0.507, 0.169, 1.521]\text{group 0} = [2,4,6,10],\ \ \mu_{0,0}=5.5,\ \sigma^2_{0,0}=8.75 \Rightarrow \hat{x} \approx [-1.183,\ -0.507,\ 0.169,\ 1.521] group 1=[1,3,0,8],  μ0,1=3, σ0,12=9.5⇒x^≈[−0.649, 0.000, −0.974, 1.622]\text{group 1} = [1,3,0,8],\ \ \mu_{0,1}=3,\ \sigma^2_{0,1}=9.5 \Rightarrow \hat{x} \approx [-0.649,\ 0.000,\ -0.974,\ 1.622]

Note c0c_0 and c1c_1 no longer both normalize to ±1\pm 1 — pooling them together means c1c_1's larger values pull the group mean up, so c0c_0's values land further below it. This is the tradeoff: GroupNorm's cross-channel pooling makes statistics less noisy (more values contribute to each μ,σ2\mu,\sigma^2) at the cost of channels influencing each other's normalization.

As a sanity check on the two special cases described above: pooling all four channels into G=1G=1 group gives μ=4.25,σ2=10.6875\mu = 4.25, \sigma^2 = 10.6875 over all 8 values. This is exactly what LayerNorm would compute for this sample (reduction set C,H,WC,H,W), confirming G=1G{=}1 GroupNorm ≡\equiv LayerNorm. The notebook reproduces every number above and both identities directly in code.

BatchNorm vs. LayerNorm vs. GroupNorm vs. InstanceNorm

BatchNormLayerNormGroupNormInstanceNorm
Averaged overbatch + spatial (N,H,WN,H,W)channels + spatial (C,H,WC,H,W)group's channels + spatial (C/G,H,WC/G,H,W)spatial only (H,WH,W)
Kept separateper channelper sampleper sample, per groupper sample, per channel
Needs running stats at inferenceyesnonono
Works at batch size 1noyesyesyes
Typical homeCNN classifiers, large batchesTransformers, RNNs (sequence models)detection/segmentation (small per-GPU batch)style transfer, GANs
Special case of GroupNorm—G=1G=1—G=CG=C

Don't confuse GroupNorm with LayerNorm-for-CNNs

Applying plain LayerNorm (G=1G=1) to a conv feature map treats all channels as one undifferentiated group — reasonable for a Transformer's feature vector, but on a CNN's feature map it can wash out the fact that different channels detect different things. GroupNorm's mid-sized groups (e.g. 32 channels per group out of 256) keep most of the batch-independence benefit while preserving more per-channel structure than G=1G=1.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
GroupNorm & InstanceNorm: Batch-I…Batch Normalization: Forw…BatchNorm & DropoutModern CNN ArchitecturesThe Transformer Architect…