Graph Attention, Pooling & Over-smoothing
A GCN treats all neighbors equally (up to a degree weight). But not every neighbor is equally informative — a Graph Attention Network (GAT) learns which neighbors to listen to. This lesson adds attention to message passing, shows how to pool node embeddings into a single graph-level prediction, and confronts the failure mode that limits GNN depth: over-smoothing.
Real-world examples: attention weights highlighting the bonds that make a molecule toxic; graph-level readout classifying whole molecules or scenes; the depth limits that keep production GNNs shallow.
Graph attention (GAT)
Instead of fixed aggregation weights, GAT computes a learned attention coefficient for each neighbor of node , then aggregates a weighted sum:
The coefficients come from a small scoring network over the transformed pair, normalized across the neighborhood with softmax:
This is exactly self-attention (from the transformers course) restricted to the graph's edges — a token attends only to its neighbors instead of all positions.
GAT vs. GCN in one line
GCN weights a neighbor by a fixed function of degree (); GAT weights it by a learned, content-dependent . Like transformers, GAT uses multiple attention heads and concatenates them for stability and expressiveness.
Readout: from nodes to a graph
Node-level GNNs output one vector per node. For graph-level tasks ("is this molecule toxic?") you need a single vector for the whole graph — a permutation-invariant readout (a.k.a. graph pooling) over all node embeddings:
It must be permutation-invariant (the graph has no node order), so sum/mean/max are the standard choices — sum preserves size information, mean is scale-stable. Hierarchical pooling methods (DiffPool, Top-K) instead coarsen the graph in stages, learning to merge nodes into clusters before the final readout.
Over-smoothing: why GNNs stay shallow
Stack too many message-passing layers and every node ends up aggregating nearly the whole graph; their embeddings converge to the same vector, destroying the distinctions you need. This is over-smoothing, and it's why most GNNs use only 2–4 layers — the opposite of the "deeper is better" instinct from CNNs.
It is the operator, not the training
Depth collapses the graph
24 nodes in two communities, each carrying a 2-D feature. Each layer is one multiplication by the GCN operator D^(-1/2)(A+I)D^(-1/2) — no weights, no nonlinearity, just the averaging. Yellow edges join the communities. Node colour is h/√degree on a fixed scale, so collapse reads as the colours converging; the raw features converge to something proportional to √degree rather than to one identical vector, which is the same statement — every node keeps its degree and loses everything else. Right: Dirichlet energy on a log axis, with the λ₂^(2k) law dashed from layer 5, where it takes over.
Strip a GCN of its weights and nonlinearities and one layer is just multiplication by . That operator has exactly, and its eigenvector is — the direction in which every node holds the same value. Every other component is multiplied by its own at every layer. Over-smoothing is not something that goes wrong during training; it is what repeated averaging is.
That also fixes the rate. Dirichlet energy measures how much node-to-node variation survives, and it shrinks by per layer. On the default graph the measured ratio walks straight onto the prediction:
| depth | measured | |
|---|---|---|
| 5 | 0.81372 | 0.82637 |
| 10 | 0.82150 | 0.82637 |
| 20 | 0.82448 | 0.82637 |
| 40 | 0.82608 | 0.82637 |
| 78 | 0.82636 | 0.82637 |
Now drag the bridge slider, because it runs against the obvious intuition. Adding edges between the two communities makes the graph mix better, which lowers , so a better-connected graph over-smooths faster. At depth 20 the surviving energy is with one bridge and with twelve: a 57× difference from adding eleven edges, with the node features untouched. More connectivity is not more information here; it is a shorter fuse.
Mitigations:
- Residual / skip connections: add the previous layer's representation back, preserving node identity. This is the α slider: APPNP's has a nonzero fixed point, so the energy plateaus instead of decaying. At , ; at it is and flat from about layer 10. One term converts geometric decay into a floor.
- Jumping knowledge: combine representations from all layers (not just the last) per node.
- Fewer layers: often the simplest, most effective fix.
- Normalization (PairNorm, etc.): explicitly keep node embeddings from collapsing together.
Receptive field vs. over-smoothing — a real tension
More layers grow the receptive field (good: reach farther neighbors) but worsen over-smoothing (bad: representations collapse). For long-range tasks this tension is fundamental, and is part of why graph transformers (global attention without repeated local averaging) have become popular.
Exercise
How does a Graph Attention Network (GAT) differ from a GCN in how it weights a node's neighbors?
Need a hint?
Fixed function of degree, or learned from the features?
Exercise
To classify a whole molecule (a graph-level task) from per-node embeddings, what must the readout function be?
Need a hint?
The graph has no node ordering.
Exercise
Why do most GNNs use only 2–4 layers rather than the dozens common in CNNs?
Need a hint?
What happens to node embeddings after many rounds of neighbor averaging?
Common mistakes
- Treating GAT attention as global. It's restricted to each node's neighbors. It does not attend across the whole graph like a transformer.
- Using a non-invariant readout. Concatenating node embeddings in some order breaks permutation invariance; pool with sum/mean/max.
- Going deep by default. Beyond ~4 layers, over-smoothing usually hurts; add skip connections or use fewer layers.
- Confusing node- and graph-level outputs. Graph-level tasks need a readout; forgetting it leaves you with per-node vectors and no graph prediction.
Key takeaways
- GAT learns content-dependent attention weights over neighbors (edge-restricted self-attention), often with multiple heads — vs. GCN's fixed degree weighting.
- Readout / graph pooling turns per-node embeddings into one graph vector with a permutation-invariant op (sum/mean/max) for graph-level tasks.
- Over-smoothing makes node embeddings converge as depth grows, capping practical GNNs at 2–4 layers.
- Fixes include skip/residual connections, jumping knowledge, normalization, and simply fewer layers; the receptive-field-vs-smoothing tension motivates graph transformers.
Related concepts
- Graph Convolutions & GraphSAGE: the fixed-weight aggregation GAT makes learnable
- Self-Attention: the attention mechanism GAT restricts to graph edges
- Pooling and Architectures: the grid analogue of graph readout/pooling