Pooling and CNN ArchitecturesOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Pooling and CNN Architectures

After convolution, we need to reduce spatial dimensions and build deeper representations. Pooling does the downsampling, and famous architectures show how to stack layers effectively.

Pooling

Pooling reduces the spatial size of feature maps, decreasing computation and providing translation invariance.

Max pooling

Takes the maximum value in each window:

MaxPool(X)i,j=max⁡m,n∈windowXi+m,j+n\text{MaxPool}(X)_{i,j} = \max_{m,n \in \text{window}} X_{i+m, j+n}
Python
import torch.nn as nn

# Max pooling: 2x2 window, stride 2 → halves spatial dimensions
pool = nn.MaxPool2d(kernel_size=2, stride=2)

# Input: [1, 64, 28, 28] → Output: [1, 64, 14, 14]
x = torch.randn(1, 64, 28, 28)
print(f"Before: {x.shape}")   # [1, 64, 28, 28]
print(f"After: {pool(x).shape}")  # [1, 64, 14, 14]

Average pooling

Takes the mean value in each window. Less common in modern architectures but used in the final layer (Global Average Pooling).

Max vs Average pooling

Max pooling preserves the strongest activation — "did this feature appear anywhere in the region?" Average pooling preserves the average activation — "how strong was this feature overall?" Max pooling is more common because it's better at detecting presence of features.

Global Average Pooling (GAP)

Reduces each feature map to a single value by averaging all pixels:

GAP(feature map)=1H×W∑i,jXi,j\text{GAP}(\text{feature map}) = \frac{1}{H \times W} \sum_{i,j} X_{i,j}

GAP replaces fully connected layers at the end of the network, drastically reducing parameters:

Python
# GAP: [1, 512, 7, 7] → [1, 512]
gap = nn.AdaptiveAvgPool2d(1)
x = torch.randn(1, 512, 7, 7)
print(f"After GAP: {gap(x).shape}")  # [1, 512, 1, 1]

Worked example

A 4×44\times4 feature map, 2×22\times2 window, stride 2 (the four non-overlapping quadrants):

X=[1324561272831041]  ⇒  Max=[6478],Avg=[3.752.252.54.0].X=\begin{bmatrix}1&3&2&4\\5&6&1&2\\7&2&8&3\\1&0&4&1\end{bmatrix} \;\Rightarrow\; \text{Max}=\begin{bmatrix}6&4\\7&8\end{bmatrix},\quad \text{Avg}=\begin{bmatrix}3.75&2.25\\2.5&4.0\end{bmatrix}.

Top-left quadrant {1,3,5,6}\{1,3,5,6\}: max =6=6, avg =1+3+5+64=3.75=\tfrac{1+3+5+6}{4}=3.75. Each quadrant collapses to one number, so 4×4→2×24\times4 \to 2\times2.

Output size formula

Pooling uses the same size formula as convolution — for window ff, stride ss, padding pp:

out=⌊n−f+2ps⌋+1\text{out} = \left\lfloor \frac{n - f + 2p}{s} \right\rfloor + 1
OperationWindowStridePaddingOutpute.g. n=28n{=}28
MaxPool2×22 \times 220⌊(n−2)/2⌋+1=n/2\lfloor(n-2)/2\rfloor+1 = n/228→1428\to14
MaxPool3×33 \times 321⌊(n−1)/2⌋+1\lfloor(n-1)/2\rfloor + 128→1428\to14
GAPH×WH \times W——1→1\to1

So a 28→14→728\to14\to7 chain of 2×22\times2 stride-2 pools halves the map twice (a final 77 with 2×22\times2 stride 2 gives ⌊5/2⌋+1=3\lfloor5/2\rfloor+1=3).

Watch the pooling window slide across the feature map — each position produces one output cell:

Pooling: sliding window downsampling

input 6×6687043098637547077828878966011722669Pool→output 3×3
position(0,0)
window2×2
stride2
output size3×3

A 2×2 pooling window slides over the 6×6 input. Max pooling keeps the largest value (preserving strong activations); average pooling takes the mean (smoother output). Stride 2 halves each spatial dimension.

Famous architectures

LeNet-5 (1998)

The first successful CNN for digit recognition:

Code
Input (32×32) → Conv(6) → Pool → Conv(16) → Pool → FC(120) → FC(84) → FC(10)

AlexNet (2012)

Won ImageNet with a huge margin. Key innovations:

  • Used ReLU instead of sigmoid/tanh
  • Dropout for regularization
  • Data augmentation (flips, crops)
  • GPU training

VGGNet (2014)

Showed that depth matters. Uses only 3×33 \times 3 filters:

Code
[Conv3-64]×2 → Pool → [Conv3-128]×2 → Pool → [Conv3-256]×3 → Pool → ...

Why only 3×33\times3? Two stacked 3×33\times3 beat one 5×55\times5. Stacking LL convolutions of size ff (stride 1) grows the receptive field linearly:

RFL=1+L (f−1).\text{RF}_L = 1 + L\,(f-1).

Each layer adds f−1f-1 to the reach (the first sees ff inputs; every layer after extends both edges by f−12\tfrac{f-1}{2}). For f=3f=3: one layer sees 33, two layers see 1+2(3−1)=51+2(3-1)=5 — the same receptive field as a single 5×55\times5. But the parameter counts differ. For CC input and CC output channels:

  • one 5×55\times5: 5⋅5⋅C2=25C25\cdot5\cdot C^2 = 25C^2
  • two 3×33\times3: 2 (3⋅3⋅C2)=18C22\,(3\cdot3\cdot C^2) = 18C^2

That's 28% fewer parameters for the same receptive field (e.g. C=64C=64: 102,400102{,}400 vs 73,72873{,}728) — and an extra ReLU in between for more non-linearity. This is the whole VGG design philosophy.

ResNet (2015)

Introduced skip connections to train very deep networks (152+ layers):

y=F(x,{Wi})+x\mathbf{y} = F(\mathbf{x}, \{W_i\}) + \mathbf{x}

The skip connection allows gradients to flow directly through the network, solving the vanishing gradient problem.

Why skip connections work

Without skip connections, gradients must flow through every layer. In a 50-layer network, the gradient is multiplied by 50 weight matrices. It can vanish or explode. Skip connections create "gradient highways" that let errors flow directly to early layers.

Python
class ResidualBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1)
        self.bn1 = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1)
        self.bn2 = nn.BatchNorm2d(channels)
    
    def forward(self, x):
        residual = x                          # save input
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += residual                       # skip connection!
        return F.relu(out)

The evolution at a glance

ArchitectureYearKey InnovationParams
LeNet1998First practical CNN60K
AlexNet2012ReLU, Dropout, GPU60M
VGG-162014Depth with 3×3 filters138M
ResNet-502015Skip connections25M
EfficientNet2019Compound scaling5M-66M

The mechanics behind the modern rows (1×1 convolutions, bottleneck blocks, depthwise separability, compound scaling) are Modern Architectures.

What the layers learn

Peeking inside a trained CNN reveals a hierarchy:

  • Layer 1: Edges, colors, simple textures
  • Layer 2: Corners, patterns, textures
  • Layer 3: Parts of objects (eyes, wheels, leaves)
  • Layer 4-5: Whole objects (faces, cars, animals)

This hierarchical feature learning is what makes CNNs powerful — they automatically discover the right representations.

Practical tip

When using a pretrained CNN, the early layers already detect useful features. You can reuse them for new tasks — the foundation of transfer learning. And the tools that show you these features (saliency maps, Grad-CAM) are in CNN Visualization & Adversarial Attacks.

Exercise

What does max pooling with a 2×2 window and stride 2 do to the spatial dimensions?

Need a hint?

Think about how many non-overlapping windows fit.

Exercise

What problem do skip connections in ResNets solve?

Need a hint?

Think about what happens to gradients in very deep networks.

Exercise

Which architecture introduced the concept of skip connections?

Need a hint?

It enabled training networks with 152+ layers.

Common mistakes

  1. Using pooling everywhere: Modern architectures prefer strided convolutions (stride=2) over max pooling. Pooling loses spatial information.
  2. Not using batch normalization: BN stabilizes training and allows higher learning rates. Always use it after conv layers.
  3. Ignoring the classifier head: GAP + a single FC layer is better than multiple FC layers. Fewer parameters, less overfitting.

Key takeaways

  • Pooling shrinks spatial size, adds translation invariance, and cuts compute.
  • Architectures evolved LeNet → AlexNet → VGG → ResNet (skip connections).
  • Skip connections let gradients bypass layers, enabling 100+ layer nets.
  • Global average pooling replaces giant fully-connected heads in modern CNNs.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Pooling and CNN ArchitecturesNeural NetworksLinear Algebra for MLCalculus for MLThe Convolution OperationModern CNN ArchitecturesTransfer LearningCNN Visualization & Adver…Gradient DescentSegmentation