GroupNorm & InstanceNorm: Batch-Independent Normalization
BatchNorm normalizes over the batch dimension, which breaks down at small batch sizes and needs running statistics for inference. LayerNorm fixes the batch-dependence by normalizing over the feature dimension of each sample independently, but for image tensors, "feature dimension" means all channels and spatial positions at once, which can be too coarse. GroupNorm and InstanceNorm sit between these two: both normalize per sample (so they need no running statistics and work at batch size 1), but they differ in how much of the channel dimension they pool together.
The shared formula
Every member of this normalization family (BatchNorm, LayerNorm, GroupNorm, InstanceNorm) computes the exact same two-step transform. Only the set of elements averaged together, , changes:
with learnable per-channel exactly as in BatchNorm. For a feature map tensor of shape (batch, channels, height, width), the four variants pick along different axes:
| Method | Reduction set (what's averaged) | Kept independent |
|---|---|---|
| BatchNorm | : all samples, all spatial positions | one stat per channel |
| LayerNorm | : all channels, all spatial positions | one stat per sample |
| GroupNorm | : channels within one group, all spatial positions | one stat per sample and group |
| InstanceNorm | : spatial positions only | one stat per sample and channel |
GroupNorm: normalize within channel groups, per sample
GroupNorm (Wu & He, 2018) splits the channels into groups of channels each, then computes one mean/variance per (sample, group) pair, pooling over that group's channels and the spatial dimensions:
Every element in group of sample is normalized with the same , then rescaled with its own channel's .
Because the statistics are computed per-sample, GroupNorm's output for one image never depends on any other image in the batch — batch size 1 works identically to batch size 256. This is exactly why it replaces BatchNorm in object detection and segmentation backbones (Mask R-CNN, etc.): high-resolution inputs mean only 1-2 images fit per GPU, too few for BatchNorm's batch statistics to be meaningful.
GroupNorm has two special cases at the extremes of :
- (one group containing all channels) — every channel pools together, which is exactly LayerNorm's reduction over .
- (each channel is its own group of size 1) — no cross-channel pooling at all, which is exactly InstanceNorm below.
Choosing G
The GroupNorm paper finds works well across a wide range of (e.g. ResNet-50's 256/512/1024/2048-channel stages), as long as stays reasonably large — too many tiny groups makes the per-group statistics noisy, just like BatchNorm gets noisy at tiny batch sizes.
InstanceNorm: normalize each channel, per sample, alone
InstanceNorm (Ulyanov et al., 2016, originally for style transfer) is the endpoint of GroupNorm: each channel of each sample is normalized using only its own spatial positions — no pooling across channels or across the batch at all:
Why this matters for style transfer and GANs: a feature channel's mean and variance across its own spatial map largely encode contrast and overall intensity — properties of "how this particular image happened to be lit or scaled," not the image's content. Subtracting and dividing by strips that per-instance contrast out, channel by channel, leaving a representation that is invariant to it. Style-transfer methods (and the AdaIN mechanism used in many GAN generators) exploit this directly: they normalize away the content image's own contrast with InstanceNorm, then re-inject a target style's mean and variance via instead of learned constants — swapping one image's "contrast signature" for another's.
InstanceNorm as contrast removal
Scaling and shifting a single channel of a single sample ( for any ) leaves InstanceNorm's output for that channel completely unchanged, because and absorb exactly that transform. That invariance is the formal version of "removes instance-specific contrast," and the notebook below verifies it numerically.
Worked trace
Take a single sample () with channels and spatial positions (2 numbers per channel), grouped into groups of 2 channels each:
| channel | values | group |
|---|---|---|
| group 0 | ||
| group 0 | ||
| group 1 | ||
| group 1 |
InstanceNorm (, each channel normalized alone over its 2 spatial values):
Every channel comes out as despite wildly different raw scales (variance 1 vs. variance 16) — exactly the contrast-removal property above: any 2-element vector normalizes to regardless of its original mean or spread.
GroupNorm (: group 0 pools ; group 1 pools , over both channels and spatial positions at once):
Note and no longer both normalize to — pooling them together means 's larger values pull the group mean up, so 's values land further below it. This is the tradeoff: GroupNorm's cross-channel pooling makes statistics less noisy (more values contribute to each ) at the cost of channels influencing each other's normalization.
As a sanity check on the two special cases described above: pooling all four channels into group gives over all 8 values. This is exactly what LayerNorm would compute for this sample (reduction set ), confirming GroupNorm LayerNorm. The notebook reproduces every number above and both identities directly in code.
BatchNorm vs. LayerNorm vs. GroupNorm vs. InstanceNorm
| BatchNorm | LayerNorm | GroupNorm | InstanceNorm | |
|---|---|---|---|---|
| Averaged over | batch + spatial () | channels + spatial () | group's channels + spatial () | spatial only () |
| Kept separate | per channel | per sample | per sample, per group | per sample, per channel |
| Needs running stats at inference | yes | no | no | no |
| Works at batch size 1 | no | yes | yes | yes |
| Typical home | CNN classifiers, large batches | Transformers, RNNs (sequence models) | detection/segmentation (small per-GPU batch) | style transfer, GANs |
| Special case of GroupNorm | — | — |
Don't confuse GroupNorm with LayerNorm-for-CNNs
Applying plain LayerNorm () to a conv feature map treats all channels as one undifferentiated group — reasonable for a Transformer's feature vector, but on a CNN's feature map it can wash out the fact that different channels detect different things. GroupNorm's mid-sized groups (e.g. 32 channels per group out of 256) keep most of the batch-independence benefit while preserving more per-channel structure than .
Related concepts
- Batch Normalization: Forward Pass & Running Stats: the batch-statistics member of this family, including the running-average mechanism GroupNorm/InstanceNorm don't need
- BatchNorm & Dropout: the lesson-level intuition for why normalization stabilizes training
- Transformer Architecture: the full LayerNorm derivation ( over the feature dimension) and why sequence models need it instead of BatchNorm
- Modern CNN Architectures: where GroupNorm replaces BatchNorm in small-batch detection/segmentation backbones