Modern CNN Architectures
Understanding why each architecture works (not just that it works) is what separates practitioners who can adapt to new problems from those who can only copy existing recipes. This lesson unpacks the key design decisions from ResNet, MobileNet, and EfficientNet that still dominate computer vision in 2025.
1×1 Convolutions: the Swiss army knife
A convolution applies a learned linear combination across channels at every spatial position, without looking at neighbouring pixels. For an input of shape and filters of size :
This is mathematically equivalent to applying a fully-connected layer independently to every spatial position's channel vector.
Three uses for convolutions:
| Use | Example | Why |
|---|---|---|
| Channel reduction | before a | Cuts FLOPs by |
| Channel expansion | after a | Restores capacity |
| Non-linear projection | Any channel-mixing | Adds expressivity without spatial mixing |
Worked parameter count
A convolution mapping 256 input channels to 256 output channels costs:
Replace this with a bottleneck: (256→64) + (64→64) + (64→256):
8.5× fewer parameters for approximately the same computation at the bottleneck. This is the key to making deep networks practical.
1×1 convolutions in the wild
You see 1×1 convolutions everywhere: the Inception module (GoogLeNet), ResNet bottleneck blocks, MobileNet's pointwise step, and the projection shortcut in ResNet when dimensions change. They are the dominant tool for controlling the channel-count/computation tradeoff.
ResNet: residual connections in depth
The degradation problem
Before ResNet (2015), adding more layers to a network consistently hurt both training and test accuracy beyond a certain depth — not just overfitting, but degradation. A 56-layer network performed worse than a 20-layer network even on training data. This is puzzling because the 56-layer network could in principle represent the 20-layer network by learning identity mappings for the extra 36 layers.
The insight: learning the identity mapping is harder than it should be. If a layer should output (unchanged), its weights must conspire to produce from a complicated nonlinear transformation. But if we give the layer a residual formulation (explicitly add back to the output), then the layer only needs to learn the residual (set all weights to zero), which is easy.
The residual block
where is a stack of layers (convolution → BN → ReLU → convolution → BN). The addition happens before the final ReLU.
Why this works mathematically: Consider the gradient flowing backward through this block. By the chain rule:
The constant is the skip path: even if (dead layers), the gradient still flows. In a 152-layer ResNet, the product of 152 Jacobians that would normally vanish is replaced by a sum of exponentially many paths — most of which go through skip connections and have well-conditioned gradients.
Bottleneck residual block
For ResNet-50 and deeper, each residual block is a bottleneck:
Input (256 ch)
↓ 1×1 conv (256→64) + BN + ReLU [channel compression]
↓ 3×3 conv (64→64) + BN + ReLU [spatial processing]
↓ 1×1 conv (64→256) + BN [channel expansion]
↓ + shortcut (identity)
↓ ReLU
Output (256 ch)
The layers bookend the expensive , reducing it to channels and restoring to afterward. Compared to two (256→256) layers, the bottleneck uses vs parameters — 17× fewer.
Projection shortcut
When the input and output of a block have different shapes (different channels or spatial size), the identity shortcut cannot be added directly. A projection shortcut uses a convolution to match dimensions:
where is a convolution that changes channel count and/or applies stride 2 to halve spatial dimensions. This is used at each stage transition in ResNet.
Convolution: a kernel slides over the input
input 6×6
feature map 4×4
The kernel multiplies its weights with the overlapping input patch and sums them into one output cell (stride 1, no padding → 6×6 becomes 4×4). The same small kernel is reused at every position — that's weight sharing.
Depthwise Separable Convolutions (MobileNet)
Standard convolution mixes channels and spatial structure simultaneously. Depthwise separable convolution (MobileNet, 2017) splits this into two steps:
- Depthwise convolution: apply one filter per input channel independently (no cross-channel mixing)
- Pointwise convolution: apply conv to mix channels
Parameter comparison for , input and output channels:
| Parameters | FLOPs (per spatial position) | |
|---|---|---|
| Standard conv | ||
| Depthwise + pointwise | ||
| Reduction | (for large ) |
MobileNet-V1 achieves ImageNet accuracy within ~1% of VGG-16 at 8× fewer FLOPs — the key to running CNNs on mobile devices.
EfficientNet: compound scaling
Prior to EfficientNet (2019), architecture improvements came from independently scaling:
- Width (more channels per layer): more features, more parameters
- Depth (more layers): more complex functions, harder to train
- Resolution (larger input images): more detail, more computation
EfficientNet (Tan & Le, 2019) showed that these three dimensions must be scaled jointly in fixed ratios to be efficient. Their compound scaling rule:
where is the overall scaling coefficient (budget) and are found by grid search such that (doubling FLOPs per doubling of ).
The intuition: higher resolution images need deeper networks to capture fine-grained patterns, and wider networks to capture more features at each level. Scaling one dimension alone hits diminishing returns quickly.
| Model | FLOPs (Billion) | Top-1 Accuracy |
|---|---|---|
| ResNet-50 | 4.1 | 76.0% |
| EfficientNet-B0 | 0.39 | 77.1% |
| EfficientNet-B7 | 37 | 84.4% |
EfficientNet-B0 achieves better accuracy than ResNet-50 with fewer FLOPs — purely from better scaling of the same building blocks.
Architecture design principles
The progression from AlexNet (2012) to modern networks reveals a consistent set of principles:
| Principle | Example | Benefit |
|---|---|---|
| Use small filters stacked | VGG 3×3 | Same RF, fewer params, more ReLUs |
| Add skip connections | ResNet | Gradient highways, train 100s of layers |
| Reduce channels before 3×3 | Bottleneck | 8-17× fewer params at same expressivity |
| Factorize spatial and channel mixing | MobileNet | 8× fewer FLOPs for mobile deployment |
| Scale width/depth/resolution jointly | EfficientNet | Pareto-efficient accuracy/compute tradeoff |
| Use BN after every conv | All modern nets | Stable training, higher LR possible |
Starting point for new tasks
For a new vision task in 2025: start with EfficientNet-B0 or ResNet-50 pretrained on ImageNet, fine-tune on your data with a small learning rate for the backbone and a larger one for the head. This gets you to a strong baseline in hours, not days.
Exercise
A 1×1 convolution takes a (H, W, 256) feature map and outputs (H, W, 64). How many learnable parameters does it have (ignoring bias)?
Need a hint?
A 1×1 conv is equivalent to a fully-connected layer applied at each spatial position independently.
Exercise
A ResNet bottleneck block processes a (H, W, 256) input with: 1×1 conv (256→64), 3×3 conv (64→64), 1×1 conv (64→256). Approximately how many parameters does this use vs. a naive 3×3 conv (256→256)?
Need a hint?
Count parameters for each layer: kernel_h × kernel_w × C_in × C_out.
Exercise
In a residual block y = F(x) + x, what is ∂y/∂x?
Need a hint?
Differentiate both the residual path F(x) and the skip path x.
Common mistakes
- Building VGG-style networks: fully-connected heads with hundreds of millions of parameters overfit heavily and are slow. Use Global Average Pooling + one FC layer instead.
- Forgetting projection shortcuts: when changing channels or stride at a stage transition, the shortcut must also change dimensions. Without this, the residual addition fails with shape mismatch.
- Removing BatchNorm from residual blocks: BN is critical for stable residual training. Each conv in the block should be followed by BN, but the shortcut path should not have BN (it would shift the identity mapping).
Key takeaways
- convolutions mix channels without spatial processing — the core tool for controlling channel count and FLOP budgets.
- ResNet's bottleneck block uses → → to process deep features with 8-17× fewer parameters.
- Skip connections solve the degradation problem by providing gradient highways and making identity learning easy.
- Depthwise separable convolutions (MobileNet) cut FLOPs 8× by factorizing channel and spatial mixing.
- Compound scaling (EfficientNet) jointly scales depth, width, and resolution for efficient accuracy-compute tradeoffs.
Related concepts
- Pooling & CNN Architectures: the foundation these architectures build on
- Transfer Learning: using pretrained ResNets/EfficientNets for new tasks
- BatchNorm & Dropout: normalization used in every modern block
- GroupNorm & InstanceNorm: the batch-size-independent normalization variants used in modern conv nets