Inductive Bias & EquivarianceOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Inductive Bias & Equivariance

The hypothesis-space stage of the ML loop asks: which family of functions can the model express? Architecture is how you answer that, and the architectures look like four unrelated topics only until you see the single idea underneath. Match the model's built-in symmetry to the symmetry in the data. Do that and four topics collapse into one, with four instantiations.

Inductive bias, precisely

An inductive bias is any assumption a model makes that isn't in the data — it's what lets a finite training set pick out a function over infinite unseen inputs. Without a bias, learning is impossible (the no free lunch idea): every function that fits the training data equally is equally preferred.

The most powerful biases in deep learning are symmetries — transformations of the input that shouldn't change the answer (or should change it in a predictable way).

Equivariance vs invariance

For a transformation gg (a shift, a permutation, a rotation) and a layer ff:

  • Invariance: f(g⋅x)=f(x)f(g \cdot x) = f(x) — the output ignores the transformation. (An image classifier should be shift-invariant: a shifted cat is still a cat.)
  • Equivariance: f(g⋅x)=g⋅f(x)f(g \cdot x) = g \cdot f(x) — the output transforms the same way the input did. (A convolution is shift-equivariant: shift the image and every feature shifts by the same amount.)
f(g⋅x)=g⋅f(x)(equivariance)f(g⋅x)=f(x)(invariance)f(g \cdot x) = g \cdot f(x) \quad \text{(equivariance)} \qquad\qquad f(g \cdot x) = f(x) \quad \text{(invariance)}

Deep nets are built from equivariant layers (convolutions, attention, message passing) and reach invariance only at the end, via pooling or a readout. Keeping equivariance through the depth is what preserves spatial/structural information.

One idea, four architectures

ArchitectureSymmetry it assumesEquivariant toWhat it shares (the "tied weights")
CNNtranslationshifting the inputthe same kernel slides over every position
RNNtime-translationshifting in timethe same recurrent weights at every step
Transformerpermutationreordering the tokensthe same attention applied to every pair
GNNpermutation over a graphrelabelling the nodesthe same message function on every edge

The shared column is the point: an inductive bias is almost always weight sharing across a symmetry group. That sharing is what buys sample efficiency — the model learns a pattern once and reuses it everywhere the symmetry says it should, instead of re-learning "edge detector" independently in all 224×224224 \times 224 positions.

Why this makes attention feel inevitable

A transformer assumes only permutation symmetry — the weakest of the four. That is exactly why it needs positional encodings: pure permutation-equivariance throws away order, so you inject position back as a signal. Read attention as "remove the strong locality bias of convolution, keep only permutation symmetry, and add position as data." It generalises the others because it assumes the least, and pays for that generality with data.

The bias/data trade-off

A stronger, correct bias needs less data; a weaker bias needs more but bends to more problems. That is the same axis as bias–variance, one level up — architectural bias instead of estimator bias.

  • Right bias, small data → wins. A CNN beats an MLP on images in the small-data regime because translation equivariance is true of images and the MLP must learn it from scratch.
  • Wrong bias → it breaks. Impose a bias the data doesn't have and you cap accuracy no matter how much data you add. A CNN on tabular data assumes a spatial locality that isn't there; a plain RNN assumes recency matters most and starves long-range dependencies (the vanishing-gradient story).
  • No bias, huge data → also wins, eventually. With enough data and compute a low-bias transformer overtakes a high-bias CNN even on images (ViT) — it learns the locality the CNN was born with. The bias was a data-efficiency shortcut, not a ceiling.

This is the moving-bottleneck story of the ML loop seen from the hypothesis-space slot: as data and compute grew, the field could afford weaker biases.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Inductive Bias & EquivarianceThe ML Project LoopVC DimensionBias-Variance Decompositi…The Convolution OperationRecurrent Neural NetworksSelf-AttentionGraphs as Data & Message …