The Convolution Operation
A CNN starts with a simple idea: instead of looking at every pixel independently, slide a small filter (kernel) across the image to detect local patterns. That "same filter everywhere" rule is a hypothesis-space choice (a built-in translation-equivariance bias) and it's the reason a CNN beats a plain MLP on images with far less data. It's the first of the four architectures in the inductive-bias story.
Real-world examples: Image classification (is this a cat?), face detection, medical imaging (tumor detection in X-rays), self-driving cars (pedestrian detection).
What is convolution?
Given an input matrix and a filter of size , the output at position is:
The filter slides across every position, computing a dot product at each step. This produces a feature map — a new image highlighting where the filter's pattern appears.
Cross-correlation vs. true convolution
Strictly, the formula above is cross-correlation: the filter is not flipped before being slid over the input. True mathematical convolution would replace with (a 180° flip). CNNs universally use cross-correlation but call it "convolution" by convention, since the filter weights are learned, the flip is absorbed into the weights and the distinction has no practical effect. PyTorch's nn.Conv2d computes cross-correlation.
Intuition
Think of a edge-detection filter. It slides across the image and outputs high values where edges exist. Each filter detects a different pattern — edges, corners, textures, or eventually complex shapes. The network learns which filters to use.
Worked example
A filter applied to a input:
Input:
1 0 1 0 1
0 1 0 1 0
1 0 1 0 1
0 1 0 1 0
1 0 1 0 1
Filter (edge detection):
1 0 -1
2 0 -2
1 0 -1
Output at position (1,1):
import numpy as np
# Input and filter
X = np.array([[1,0,1,0,1],
[0,1,0,1,0],
[1,0,1,0,1],
[0,1,0,1,0],
[1,0,1,0,1]], dtype=float)
K = np.array([[1, 0, -1],
[2, 0, -2],
[1, 0, -1]], dtype=float)
# Manual convolution at position (1,1)
output_11 = np.sum(X[1:4, 1:4] * K)
print(f"Output at (1,1): {output_11}") # 0
# Full convolution using scipy
from scipy.signal import convolve2d
output = convolve2d(X, np.flip(K), mode='valid') # flip K: convolve2d does true convolution
print(f"Full output:\n{output}")
Every valid position gives here — the checkerboard has no vertical edges for this vertical-edge filter to fire on. A non-uniform input makes the mechanism visible. Take a input with the same filter (output size , so ):
Top-left window (rows 0–2, cols 0–2) — each row contributes (left right) weighted by :
Sliding over all four positions gives the feature map — now the filter's response varies with the local pattern.
Convolution: a kernel slides over the input
input 6×6
feature map 4×4
The kernel multiplies its weights with the overlapping input patch and sums them into one output cell (stride 1, no padding → 6×6 becomes 4×4). The same small kernel is reused at every position — that's weight sharing.
Key parameters
Filter size
Common choices: (most common), , . Smaller filters capture finer details. Modern networks use mostly filters — stacking two filters has the same receptive field as one but with fewer parameters and more non-linearity.
Stride
How many pixels the filter moves at each step. Stride means every position. Stride means skip every other position, halving the output size.
Padding
Adding zeros around the border. "Same" padding preserves the spatial size. "Valid" padding uses no padding (output shrinks).
Output size formula:
where = input size, = filter size, = padding, = stride.
Where it comes from. Pad first, so the working width is . The filter's left edge can sit at column and slide right as long as the filter still fits — its last valid left-edge position is . With stride the valid positions are up to , which is positions (the counts position ). That count is the output size.
Worked demonstrations:
| out | |||||
|---|---|---|---|---|---|
| 5 | 3 | 0 | 1 | 3 | |
| 7 | 3 | 1 | 2 | 4 | |
| 32 | 3 | 1 | 1 | 32 (same padding) | |
| 28 | 5 | 0 | 1 | 24 |
So the input with a valid filter above produces a feature map.
Quick reference
| Padding | Output Size |
|---|---|
| Valid () | |
| Same () |
Why convolution is powerful
Weight sharing
The same filter is used across the entire image. A filter has only 9 parameters regardless of image size. A fully connected layer for a image would need parameters.
Translation equivariance
If a cat moves in the image, the feature map shifts but the same features are detected. The network doesn't need to learn "cat at top-left" and "cat at bottom-right" separately.
Parameter count
Each of the filters has weights plus one bias, so a conv layer has parameters. For a layer with filters: — that's weights plus biases (which is why the PyTorch layer below reports 1792). Compare this to a fully connected layer: millions.
Multi-channel convolution
Real images have 3 channels (RGB). A conv layer takes channels and produces channels:
- Each of the filters has depth
- Each filter produces one 2D feature map
- Stack the feature maps into the output
Input: H × W × C_in
Filter: f × f × C_in (one of C_out filters)
Output: H' × W' × C_out
import torch.nn as nn
# PyTorch conv layer: 3 input channels → 64 output channels
conv = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, padding=1)
print(f"Weight shape: {conv.weight.shape}") # [64, 3, 3, 3]
print(f"Parameters: {sum(p.numel() for p in conv.parameters())}") # 1792
Building blocks of a CNN layer
A typical conv layer is: Convolution → Activation (ReLU) → BatchNorm (optional)
The ReLU adds non-linearity, allowing the network to learn complex patterns:
# Typical CNN block
class ConvBlock(nn.Module):
def __init__(self, in_ch, out_ch):
super().__init__()
self.block = nn.Sequential(
nn.Conv2d(in_ch, out_ch, 3, padding=1),
nn.BatchNorm2d(out_ch),
nn.ReLU(inplace=True)
)
def forward(self, x):
return self.block(x)
Exercise
A 32×32 input with a 5×5 filter, stride 2, and padding 2 produces what output size?
Need a hint?
Use the formula: out = floor((n - f + 2p) / s) + 1.
Exercise
A conv layer has 64 input channels, 128 output channels, and 3×3 filters. How many parameters does it have?
Need a hint?
Parameters = C_in × C_out × f × f + C_out (bias).
Exercise
Two stacked 3×3 conv layers have the same receptive field as one layer of what size?
Need a hint?
Each layer adds to the receptive field.
Common mistakes
- Forgetting to add padding: Without "same" padding, the spatial dimensions shrink at every layer. After 5 conv layers, a input might become .
- Using too many channels early: Start with 32 or 64 channels, not 512. The first layers detect simple features (edges, textures) that don't need many channels.
- Ignoring the receptive field: Each output pixel only "sees" a small patch of the input. Stacking layers increases the receptive field.
Key takeaways
- Convolution slides a small filter over the input, producing feature maps.
- Weight sharing + translation equivariance make CNNs efficient and shift-robust.
- Output size ; padding preserves spatial size.
- A filter spans all input channels; a layer learns
C_outof them.
Related concepts
- Pooling & Architectures: stacking convolutions into real networks
- What is a Neuron?: the neuron that convolution specializes
- Self-Attention: the attention-based alternative now used for vision
- Inductive Bias & Equivariance: convolution as translation equivariance, and the one idea shared by CNNs, RNNs, transformers, and GNNs