Graph Attention, Pooling & Over-smoothingOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Graph Attention, Pooling & Over-smoothing

A GCN treats all neighbors equally (up to a degree weight). But not every neighbor is equally informative — a Graph Attention Network (GAT) learns which neighbors to listen to. This lesson adds attention to message passing, shows how to pool node embeddings into a single graph-level prediction, and confronts the failure mode that limits GNN depth: over-smoothing.

Real-world examples: attention weights highlighting the bonds that make a molecule toxic; graph-level readout classifying whole molecules or scenes; the depth limits that keep production GNNs shallow.

Graph attention (GAT)

Instead of fixed aggregation weights, GAT computes a learned attention coefficient αvu\alpha_{vu} for each neighbor uu of node vv, then aggregates a weighted sum:

hv′=σ(∑u∈N(v)αvu Whu)\mathbf{h}_v' = \sigma\Big(\sum_{u \in \mathcal{N}(v)} \alpha_{vu}\, \mathbf{W}\mathbf{h}_u\Big)

The coefficients come from a small scoring network aa over the transformed pair, normalized across the neighborhood with softmax:

αvu=exp⁡(LeakyReLU(a⊤[Whv ∥ Whu]))∑w∈N(v)exp⁡(LeakyReLU(a⊤[Whv ∥ Whw]))\alpha_{vu} = \frac{\exp\big(\text{LeakyReLU}(a^{\top}[\mathbf{W}\mathbf{h}_v \,\Vert\, \mathbf{W}\mathbf{h}_u])\big)}{\sum_{w \in \mathcal{N}(v)}\exp\big(\text{LeakyReLU}(a^{\top}[\mathbf{W}\mathbf{h}_v \,\Vert\, \mathbf{W}\mathbf{h}_w])\big)}

This is exactly self-attention (from the transformers course) restricted to the graph's edges — a token attends only to its neighbors instead of all positions.

GAT vs. GCN in one line

GCN weights a neighbor by a fixed function of degree (1/dvdu1/\sqrt{d_v d_u}); GAT weights it by a learned, content-dependent αvu\alpha_{vu}. Like transformers, GAT uses multiple attention heads and concatenates them for stability and expressiveness.

Readout: from nodes to a graph

Node-level GNNs output one vector per node. For graph-level tasks ("is this molecule toxic?") you need a single vector for the whole graph — a permutation-invariant readout (a.k.a. graph pooling) over all node embeddings:

hG=READOUT({hv:v∈V})=e.g. ∑v∈Vhv   or   meanv hv\mathbf{h}_G = \text{READOUT}\big(\{\mathbf{h}_v : v \in V\}\big) = \text{e.g. } \sum_{v\in V}\mathbf{h}_v \;\text{ or }\; \text{mean}_v\,\mathbf{h}_v

It must be permutation-invariant (the graph has no node order), so sum/mean/max are the standard choices — sum preserves size information, mean is scale-stable. Hierarchical pooling methods (DiffPool, Top-K) instead coarsen the graph in stages, learning to merge nodes into clusters before the final readout.

Over-smoothing: why GNNs stay shallow

Stack too many message-passing layers and every node ends up aggregating nearly the whole graph; their embeddings converge to the same vector, destroying the distinctions you need. This is over-smoothing, and it's why most GNNs use only 2–4 layers — the opposite of the "deeper is better" instinct from CNNs.

as k→∞:hv(k)→(nearly) the same vector for all v\text{as } k \to \infty:\quad \mathbf{h}_v^{(k)} \to \text{(nearly) the same vector for all } v

It is the operator, not the training

Depth collapses the graph

community Acommunity BDirichlet energy (log) — dashed is λ₂^2k from layer 5020 layers
top eigenvalue1.0000
second eigenvalue0.9090
predicted decay / layer0.8264
measured decay / layer0.7953
energy left15.51%
spread of h/√degree1.951

24 nodes in two communities, each carrying a 2-D feature. Each layer is one multiplication by the GCN operator D^(-1/2)(A+I)D^(-1/2) — no weights, no nonlinearity, just the averaging. Yellow edges join the communities. Node colour is h/√degree on a fixed scale, so collapse reads as the colours converging; the raw features converge to something proportional to √degree rather than to one identical vector, which is the same statement — every node keeps its degree and loses everything else. Right: Dirichlet energy on a log axis, with the λ₂^(2k) law dashed from layer 5, where it takes over.

Strip a GCN of its weights and nonlinearities and one layer is just multiplication by P=D−1/2(A+I)D−1/2P = D^{-1/2}(A+I)D^{-1/2}. That operator has λ1=1\lambda_1 = 1 exactly, and its eigenvector is D1/21D^{1/2}\mathbf{1} — the direction in which every node holds the same value. Every other component is multiplied by its own λi<1\lambda_i < 1 at every layer. Over-smoothing is not something that goes wrong during training; it is what repeated averaging is.

That also fixes the rate. Dirichlet energy E(H)=∑(u,v)∈E∥hu/du−hv/dv∥2E(H) = \sum_{(u,v)\in E}\lVert h_u/\sqrt{d_u} - h_v/\sqrt{d_v}\rVert^2 measures how much node-to-node variation survives, and it shrinks by λ22\lambda_2^2 per layer. On the default graph the measured ratio walks straight onto the prediction:

depth kkmeasured Ek+1/EkE_{k+1}/E_kλ22\lambda_2^2
50.813720.82637
100.821500.82637
200.824480.82637
400.826080.82637
780.826360.82637

Now drag the bridge slider, because it runs against the obvious intuition. Adding edges between the two communities makes the graph mix better, which lowers λ2\lambda_2, so a better-connected graph over-smooths faster. At depth 20 the surviving energy is 0.5040.504 with one bridge and 0.00880.0088 with twelve: a 57× difference from adding eleven edges, with the node features untouched. More connectivity is not more information here; it is a shorter fuse.

Mitigations:

  • Residual / skip connections: add the previous layer's representation back, preserving node identity. This is the α slider: APPNP's H←(1−α)PH+αH0H \leftarrow (1-\alpha)PH + \alpha H_0 has a nonzero fixed point, so the energy plateaus instead of decaying. At α=0\alpha = 0, E30/E0=9.6×10−4E_{30}/E_0 = 9.6\times10^{-4}; at α=0.1\alpha = 0.1 it is 1.1×10−11.1\times10^{-1} and flat from about layer 10. One term converts geometric decay into a floor.
  • Jumping knowledge: combine representations from all layers (not just the last) per node.
  • Fewer layers: often the simplest, most effective fix.
  • Normalization (PairNorm, etc.): explicitly keep node embeddings from collapsing together.

Receptive field vs. over-smoothing — a real tension

More layers grow the receptive field (good: reach farther neighbors) but worsen over-smoothing (bad: representations collapse). For long-range tasks this tension is fundamental, and is part of why graph transformers (global attention without repeated local averaging) have become popular.

Exercise

How does a Graph Attention Network (GAT) differ from a GCN in how it weights a node's neighbors?

Need a hint?

Fixed function of degree, or learned from the features?

Exercise

To classify a whole molecule (a graph-level task) from per-node embeddings, what must the readout function be?

Need a hint?

The graph has no node ordering.

Exercise

Why do most GNNs use only 2–4 layers rather than the dozens common in CNNs?

Need a hint?

What happens to node embeddings after many rounds of neighbor averaging?

Common mistakes

  1. Treating GAT attention as global. It's restricted to each node's neighbors. It does not attend across the whole graph like a transformer.
  2. Using a non-invariant readout. Concatenating node embeddings in some order breaks permutation invariance; pool with sum/mean/max.
  3. Going deep by default. Beyond ~4 layers, over-smoothing usually hurts; add skip connections or use fewer layers.
  4. Confusing node- and graph-level outputs. Graph-level tasks need a readout; forgetting it leaves you with per-node vectors and no graph prediction.

Key takeaways

  • GAT learns content-dependent attention weights αvu\alpha_{vu} over neighbors (edge-restricted self-attention), often with multiple heads — vs. GCN's fixed degree weighting.
  • Readout / graph pooling turns per-node embeddings into one graph vector with a permutation-invariant op (sum/mean/max) for graph-level tasks.
  • Over-smoothing makes node embeddings converge as depth grows, capping practical GNNs at 2–4 layers.
  • Fixes include skip/residual connections, jumping knowledge, normalization, and simply fewer layers; the receptive-field-vs-smoothing tension motivates graph transformers.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Graph Attention, Pooling & Over-s…Neural NetworksGraph Convolutions & Grap…Self-AttentionPooling and CNN Architect…