Modern CNN ArchitecturesOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Modern CNN Architectures

Understanding why each architecture works (not just that it works) is what separates practitioners who can adapt to new problems from those who can only copy existing recipes. This lesson unpacks the key design decisions from ResNet, MobileNet, and EfficientNet that still dominate computer vision in 2025.

1×1 Convolutions: the Swiss army knife

A 1×11 \times 1 convolution applies a learned linear combination across channels at every spatial position, without looking at neighbouring pixels. For an input of shape (H,W,Cin)(H, W, C_{in}) and CoutC_{out} filters of size 1×1×Cin1 \times 1 \times C_{in}:

outputh,w,k=∑c=1CinWk,c⋅inputh,w,c+bk\text{output}_{h,w,k} = \sum_{c=1}^{C_{in}} W_{k,c} \cdot \text{input}_{h,w,c} + b_k

This is mathematically equivalent to applying a fully-connected layer independently to every spatial position's channel vector.

Three uses for 1×11 \times 1 convolutions:

UseExampleWhy
Channel reduction256→64256 \to 64 before a 3×33\times3Cuts FLOPs by 4×4\times
Channel expansion64→25664 \to 256 after a 3×33\times3Restores capacity
Non-linear projectionAny channel-mixingAdds expressivity without spatial mixing

Worked parameter count

A 3×33\times3 convolution mapping 256 input channels to 256 output channels costs:

3×3⏟spatial×256⏟in×256⏟out=589,824 parameters\underbrace{3 \times 3}_{\text{spatial}} \times \underbrace{256}_{\text{in}} \times \underbrace{256}_{\text{out}} = 589{,}824 \text{ parameters}

Replace this with a bottleneck: 1×11\times1 (256→64) + 3×33\times3 (64→64) + 1×11\times1 (64→256):

(1×256×64)+(9×64×64)+(1×64×256)=16,384+36,864+16,384=69,632(1 \times 256 \times 64) + (9 \times 64 \times 64) + (1 \times 64 \times 256) = 16{,}384 + 36{,}864 + 16{,}384 = 69{,}632

8.5× fewer parameters for approximately the same computation at the bottleneck. This is the key to making deep networks practical.

1×1 convolutions in the wild

You see 1×1 convolutions everywhere: the Inception module (GoogLeNet), ResNet bottleneck blocks, MobileNet's pointwise step, and the projection shortcut in ResNet when dimensions change. They are the dominant tool for controlling the channel-count/computation tradeoff.

ResNet: residual connections in depth

The degradation problem

Before ResNet (2015), adding more layers to a network consistently hurt both training and test accuracy beyond a certain depth — not just overfitting, but degradation. A 56-layer network performed worse than a 20-layer network even on training data. This is puzzling because the 56-layer network could in principle represent the 20-layer network by learning identity mappings for the extra 36 layers.

The insight: learning the identity mapping is harder than it should be. If a layer should output x\mathbf{x} (unchanged), its weights must conspire to produce x\mathbf{x} from a complicated nonlinear transformation. But if we give the layer a residual formulation (explicitly add x\mathbf{x} back to the output), then the layer only needs to learn the residual F(x)=0F(\mathbf{x}) = 0 (set all weights to zero), which is easy.

The residual block

y=F(x)+x\mathbf{y} = F(\mathbf{x}) + \mathbf{x}

where F(x)F(\mathbf{x}) is a stack of layers (convolution → BN → ReLU → convolution → BN). The addition happens before the final ReLU.

Why this works mathematically: Consider the gradient flowing backward through this block. By the chain rule:

∂L∂x=∂L∂y⋅∂y∂x=∂L∂y⋅(1+∂F∂x)\frac{\partial \mathcal{L}}{\partial \mathbf{x}} = \frac{\partial \mathcal{L}}{\partial \mathbf{y}} \cdot \frac{\partial \mathbf{y}}{\partial \mathbf{x}} = \frac{\partial \mathcal{L}}{\partial \mathbf{y}} \cdot \left(1 + \frac{\partial F}{\partial \mathbf{x}}\right)

The constant 11 is the skip path: even if ∂F/∂x≈0\partial F / \partial \mathbf{x} \approx 0 (dead layers), the gradient still flows. In a 152-layer ResNet, the product of 152 Jacobians that would normally vanish is replaced by a sum of exponentially many paths — most of which go through skip connections and have well-conditioned gradients.

Bottleneck residual block

For ResNet-50 and deeper, each residual block is a bottleneck:

Code
Input (256 ch)
    ↓ 1×1 conv (256→64)  + BN + ReLU    [channel compression]
    ↓ 3×3 conv (64→64)   + BN + ReLU    [spatial processing]
    ↓ 1×1 conv (64→256)  + BN           [channel expansion]
    ↓ + shortcut (identity)
    ↓ ReLU
Output (256 ch)

The 1×11\times1 layers bookend the expensive 3×33\times3, reducing it to 6464 channels and restoring to 256256 afterward. Compared to two 3×33\times3 (256→256) layers, the bottleneck uses 69,63269{,}632 vs 1,179,6481{,}179{,}648 parameters — 17× fewer.

Projection shortcut

When the input and output of a block have different shapes (different channels or spatial size), the identity shortcut cannot be added directly. A projection shortcut uses a 1×11\times1 convolution to match dimensions:

y=F(x)+Ws x\mathbf{y} = F(\mathbf{x}) + W_s\,\mathbf{x}

where WsW_s is a 1×11\times1 convolution that changes channel count and/or applies stride 2 to halve spatial dimensions. This is used at each stage transition in ResNet.

Convolution: a kernel slides over the input

input 6×6

221767223788212787221687112876212877

feature map 4×4

1
position(0,0)
output value1.0

The kernel multiplies its weights with the overlapping input patch and sums them into one output cell (stride 1, no padding → 6×6 becomes 4×4). The same small kernel is reused at every position — that's weight sharing.

Depthwise Separable Convolutions (MobileNet)

Standard convolution mixes channels and spatial structure simultaneously. Depthwise separable convolution (MobileNet, 2017) splits this into two steps:

  1. Depthwise convolution: apply one K×KK\times K filter per input channel independently (no cross-channel mixing)
  2. Pointwise convolution: apply 1×11\times1 conv to mix channels

Parameter comparison for K=3K=3, CC input and CC output channels:

ParametersFLOPs (per spatial position)
Standard 3×33\times3 conv9C29C^29C29C^2
Depthwise + pointwise9C+C29C + C^29C+C29C + C^2
Reduction≈8×\approx 8\times (for large CC)≈8×\approx 8\times

MobileNet-V1 achieves ImageNet accuracy within ~1% of VGG-16 at 8× fewer FLOPs — the key to running CNNs on mobile devices.

EfficientNet: compound scaling

Prior to EfficientNet (2019), architecture improvements came from independently scaling:

  • Width (more channels per layer): more features, more parameters
  • Depth (more layers): more complex functions, harder to train
  • Resolution (larger input images): more detail, more computation

EfficientNet (Tan & Le, 2019) showed that these three dimensions must be scaled jointly in fixed ratios to be efficient. Their compound scaling rule:

depth: d=αϕ,width: w=βϕ,resolution: r=γϕ\text{depth:} \ d = \alpha^\phi, \quad \text{width:} \ w = \beta^\phi, \quad \text{resolution:} \ r = \gamma^\phi

where ϕ\phi is the overall scaling coefficient (budget) and α,β,γ\alpha, \beta, \gamma are found by grid search such that α⋅β2⋅γ2≈2\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2 (doubling FLOPs per doubling of ϕ\phi).

The intuition: higher resolution images need deeper networks to capture fine-grained patterns, and wider networks to capture more features at each level. Scaling one dimension alone hits diminishing returns quickly.

ModelFLOPs (Billion)Top-1 Accuracy
ResNet-504.176.0%
EfficientNet-B00.3977.1%
EfficientNet-B73784.4%

EfficientNet-B0 achieves better accuracy than ResNet-50 with 10×10\times fewer FLOPs — purely from better scaling of the same building blocks.

Architecture design principles

The progression from AlexNet (2012) to modern networks reveals a consistent set of principles:

PrincipleExampleBenefit
Use small filters stackedVGG 3×3Same RF, fewer params, more ReLUs
Add skip connectionsResNetGradient highways, train 100s of layers
Reduce channels before 3×3Bottleneck8-17× fewer params at same expressivity
Factorize spatial and channel mixingMobileNet8× fewer FLOPs for mobile deployment
Scale width/depth/resolution jointlyEfficientNetPareto-efficient accuracy/compute tradeoff
Use BN after every convAll modern netsStable training, higher LR possible

Starting point for new tasks

For a new vision task in 2025: start with EfficientNet-B0 or ResNet-50 pretrained on ImageNet, fine-tune on your data with a small learning rate for the backbone and a larger one for the head. This gets you to a strong baseline in hours, not days.

Exercise

A 1×1 convolution takes a (H, W, 256) feature map and outputs (H, W, 64). How many learnable parameters does it have (ignoring bias)?

Need a hint?

A 1×1 conv is equivalent to a fully-connected layer applied at each spatial position independently.

Exercise

A ResNet bottleneck block processes a (H, W, 256) input with: 1×1 conv (256→64), 3×3 conv (64→64), 1×1 conv (64→256). Approximately how many parameters does this use vs. a naive 3×3 conv (256→256)?

Need a hint?

Count parameters for each layer: kernel_h × kernel_w × C_in × C_out.

Exercise

In a residual block y = F(x) + x, what is ∂y/∂x?

Need a hint?

Differentiate both the residual path F(x) and the skip path x.

Common mistakes

  1. Building VGG-style networks: fully-connected heads with hundreds of millions of parameters overfit heavily and are slow. Use Global Average Pooling + one FC layer instead.
  2. Forgetting projection shortcuts: when changing channels or stride at a stage transition, the shortcut must also change dimensions. Without this, the residual addition fails with shape mismatch.
  3. Removing BatchNorm from residual blocks: BN is critical for stable residual training. Each conv in the block should be followed by BN, but the shortcut path should not have BN (it would shift the identity mapping).

Key takeaways

  • 1×11\times1 convolutions mix channels without spatial processing — the core tool for controlling channel count and FLOP budgets.
  • ResNet's bottleneck block uses 1×11\times1 → 3×33\times3 → 1×11\times1 to process deep features with 8-17× fewer parameters.
  • Skip connections solve the degradation problem by providing gradient highways and making identity learning easy.
  • Depthwise separable convolutions (MobileNet) cut FLOPs 8× by factorizing channel and spatial mixing.
  • Compound scaling (EfficientNet) jointly scales depth, width, and resolution for efficient accuracy-compute tradeoffs.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Modern CNN ArchitecturesNeural NetworksLinear Algebra for MLCalculus for MLPooling and CNN Architect…Transfer LearningBatchNorm & DropoutBackbones in PracticeGroupNorm & InstanceNorm:…