The Convolution OperationOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

The Convolution Operation

A CNN starts with a simple idea: instead of looking at every pixel independently, slide a small filter (kernel) across the image to detect local patterns. That "same filter everywhere" rule is a hypothesis-space choice (a built-in translation-equivariance bias) and it's the reason a CNN beats a plain MLP on images with far less data. It's the first of the four architectures in the inductive-bias story.

Deep dive
The ML Project Loop
Includes runnable notebook

Real-world examples: Image classification (is this a cat?), face detection, medical imaging (tumor detection in X-rays), self-driving cars (pedestrian detection).

What is convolution?

Given an input matrix XX and a filter KK of size f×ff \times f, the output at position (i,j)(i, j) is:

(X⋆K)i,j=∑m=0f−1∑n=0f−1Xi+m,  j+n⋅Km,n(X \star K)_{i,j} = \sum_{m=0}^{f-1} \sum_{n=0}^{f-1} X_{i+m,\; j+n} \cdot K_{m,n}

The filter slides across every position, computing a dot product at each step. This produces a feature map — a new image highlighting where the filter's pattern appears.

Cross-correlation vs. true convolution

Strictly, the formula above is cross-correlation: the filter is not flipped before being slid over the input. True mathematical convolution would replace Km,nK_{m,n} with Kf−1−m,  f−1−nK_{f-1-m,\;f-1-n} (a 180° flip). CNNs universally use cross-correlation but call it "convolution" by convention, since the filter weights are learned, the flip is absorbed into the weights and the distinction has no practical effect. PyTorch's nn.Conv2d computes cross-correlation.

Intuition

Think of a 3×33 \times 3 edge-detection filter. It slides across the image and outputs high values where edges exist. Each filter detects a different pattern — edges, corners, textures, or eventually complex shapes. The network learns which filters to use.

Worked example

A 3×33 \times 3 filter applied to a 5×55 \times 5 input:

Input:

Code
1  0  1  0  1
0  1  0  1  0
1  0  1  0  1
0  1  0  1  0
1  0  1  0  1

Filter (edge detection):

Code
 1  0 -1
 2  0 -2
 1  0 -1

Output at position (1,1):

1(1)+0(0)+1(−1)+0(2)+1(0)+0(−2)+1(1)+0(0)+1(−1)=01(1) + 0(0) + 1(-1) + 0(2) + 1(0) + 0(-2) + 1(1) + 0(0) + 1(-1) = 0
Python
import numpy as np

# Input and filter
X = np.array([[1,0,1,0,1],
              [0,1,0,1,0],
              [1,0,1,0,1],
              [0,1,0,1,0],
              [1,0,1,0,1]], dtype=float)

K = np.array([[1, 0, -1],
              [2, 0, -2],
              [1, 0, -1]], dtype=float)

# Manual convolution at position (1,1)
output_11 = np.sum(X[1:4, 1:4] * K)
print(f"Output at (1,1): {output_11}")  # 0

# Full convolution using scipy
from scipy.signal import convolve2d
output = convolve2d(X, np.flip(K), mode='valid')   # flip K: convolve2d does true convolution
print(f"Full output:\n{output}")

Every valid position gives 00 here — the checkerboard has no vertical edges for this vertical-edge filter to fire on. A non-uniform input makes the mechanism visible. Take a 4×44\times4 input with the same filter (output size ⌊(4−3)/1⌋+1=2\lfloor(4-3)/1\rfloor+1 = 2, so 2×22\times2):

X=[1230012330122301],K=[10−120−210−1].X=\begin{bmatrix}1&2&3&0\\0&1&2&3\\3&0&1&2\\2&3&0&1\end{bmatrix}, \qquad K=\begin{bmatrix}1&0&-1\\2&0&-2\\1&0&-1\end{bmatrix}.

Top-left window (rows 0–2, cols 0–2) — each row contributes (left −- right) weighted by 1,2,11,2,1:

(1−3)⋅1+(0−2)⋅2+(3−1)⋅1=−2−4+2=−4.(1{-}3)\cdot 1 + (0{-}2)\cdot 2 + (3{-}1)\cdot 1 = -2 - 4 + 2 = -4.

Sliding over all four positions gives the feature map [−4−44−4]\begin{bmatrix}-4&-4\\4&-4\end{bmatrix} — now the filter's response varies with the local pattern.

Convolution: a kernel slides over the input

input 6×6

221767223788212787221687112876212877

feature map 4×4

1
position(0,0)
output value1.0

The kernel multiplies its weights with the overlapping input patch and sums them into one output cell (stride 1, no padding → 6×6 becomes 4×4). The same small kernel is reused at every position — that's weight sharing.

Key parameters

Filter size

Common choices: 3×33 \times 3 (most common), 5×55 \times 5, 7×77 \times 7. Smaller filters capture finer details. Modern networks use mostly 3×33 \times 3 filters — stacking two 3×33 \times 3 filters has the same receptive field as one 5×55 \times 5 but with fewer parameters and more non-linearity.

Stride

How many pixels the filter moves at each step. Stride s=1s = 1 means every position. Stride s=2s = 2 means skip every other position, halving the output size.

Padding

Adding zeros around the border. "Same" padding preserves the spatial size. "Valid" padding uses no padding (output shrinks).

Output size formula:

out=⌊n−f+2ps⌋+1\text{out} = \left\lfloor \frac{n - f + 2p}{s} \right\rfloor + 1

where nn = input size, ff = filter size, pp = padding, ss = stride.

Where it comes from. Pad first, so the working width is n+2pn+2p. The filter's left edge can sit at column 00 and slide right as long as the filter still fits — its last valid left-edge position is n+2p−fn+2p-f. With stride ss the valid positions are 0,s,2s,…0, s, 2s, \dots up to n+2p−fn+2p-f, which is ⌊n+2p−fs⌋+1\lfloor\frac{n+2p-f}{s}\rfloor + 1 positions (the +1+1 counts position 00). That count is the output size.

Worked demonstrations:

nnffppss⌊(n−f+2p)/s⌋+1\lfloor(n-f+2p)/s\rfloor+1out
5301⌊2/1⌋+1\lfloor 2/1\rfloor+13
7312⌊6/2⌋+1\lfloor 6/2\rfloor+14
32311⌊32/1⌋+1\lfloor 32/1\rfloor+132 (same padding)
28501⌊24/1⌋+1\lfloor 24/1\rfloor+124

So the 5×55\times5 input with a 3×33\times3 valid filter above produces a 3×33\times3 feature map.

Quick reference

PaddingOutput Size
Valid (p=0p=0)⌊(n−f)/s⌋+1\lfloor(n-f)/s\rfloor + 1
Same (p=(f−1)/2p=(f-1)/2)⌈n/s⌉\lceil n/s \rceil

Why convolution is powerful

Weight sharing

The same filter is used across the entire image. A 3×33 \times 3 filter has only 9 parameters regardless of image size. A fully connected layer for a 256×256256 \times 256 image would need 2562×2562256^2 \times 256^2 parameters.

Translation equivariance

If a cat moves in the image, the feature map shifts but the same features are detected. The network doesn't need to learn "cat at top-left" and "cat at bottom-right" separately.

Parameter count

Each of the CoutC_{\text{out}} filters has Cin×f×fC_{\text{in}}\times f\times f weights plus one bias, so a conv layer has Cout (Cin f2+1)C_{\text{out}}\,(C_{\text{in}}\,f^2 + 1) parameters. For a 3→643 \to 64 layer with 3×33 \times 3 filters: 64 (3×9+1)=64×28=1,79264\,(3\times 9 + 1) = 64\times 28 = \mathbf{1{,}792} — that's 3×64×9=1,7283\times64\times9 = 1{,}728 weights plus 6464 biases (which is why the PyTorch layer below reports 1792). Compare this to a fully connected layer: millions.

Multi-channel convolution

Real images have 3 channels (RGB). A conv layer takes CinC_{\text{in}} channels and produces CoutC_{\text{out}} channels:

  • Each of the CoutC_{\text{out}} filters has depth CinC_{\text{in}}
  • Each filter produces one 2D feature map
  • Stack the CoutC_{\text{out}} feature maps into the output
Code
Input:  H × W × C_in
Filter: f × f × C_in  (one of C_out filters)
Output: H' × W' × C_out
Python
import torch.nn as nn

# PyTorch conv layer: 3 input channels → 64 output channels
conv = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, padding=1)
print(f"Weight shape: {conv.weight.shape}")  # [64, 3, 3, 3]
print(f"Parameters: {sum(p.numel() for p in conv.parameters())}")  # 1792

Building blocks of a CNN layer

A typical conv layer is: Convolution → Activation (ReLU) → BatchNorm (optional)

The ReLU adds non-linearity, allowing the network to learn complex patterns:

ReLU(x)=max⁡(0,x)\text{ReLU}(x) = \max(0, x)
Python
# Typical CNN block
class ConvBlock(nn.Module):
    def __init__(self, in_ch, out_ch):
        super().__init__()
        self.block = nn.Sequential(
            nn.Conv2d(in_ch, out_ch, 3, padding=1),
            nn.BatchNorm2d(out_ch),
            nn.ReLU(inplace=True)
        )
    
    def forward(self, x):
        return self.block(x)

Exercise

A 32×32 input with a 5×5 filter, stride 2, and padding 2 produces what output size?

Need a hint?

Use the formula: out = floor((n - f + 2p) / s) + 1.

Exercise

A conv layer has 64 input channels, 128 output channels, and 3×3 filters. How many parameters does it have?

Need a hint?

Parameters = C_in × C_out × f × f + C_out (bias).

Exercise

Two stacked 3×3 conv layers have the same receptive field as one layer of what size?

Need a hint?

Each layer adds to the receptive field.

Common mistakes

  1. Forgetting to add padding: Without "same" padding, the spatial dimensions shrink at every layer. After 5 conv layers, a 32×3232 \times 32 input might become 22×2222 \times 22.
  2. Using too many channels early: Start with 32 or 64 channels, not 512. The first layers detect simple features (edges, textures) that don't need many channels.
  3. Ignoring the receptive field: Each output pixel only "sees" a small patch of the input. Stacking layers increases the receptive field.

Key takeaways

  • Convolution slides a small filter over the input, producing feature maps.
  • Weight sharing + translation equivariance make CNNs efficient and shift-robust.
  • Output size =⌊(W−F+2P)/S⌋+1=\lfloor (W-F+2P)/S\rfloor + 1; padding preserves spatial size.
  • A filter spans all input channels; a layer learns C_out of them.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
The Convolution OperationNeural NetworksLinear Algebra for MLCalculus for MLWhat is a Neuron?Pooling and CNN Architect…Self-AttentionCNN Visualization & Adver…Object DetectionGraphs as Data & Message …Inductive Bias & Equivari…The ML Project Loop