Backbones in PracticeOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Backbones in Practice

Every detection and segmentation model has two parts: a backbone that converts raw pixels into rich feature representations, and one or more heads that consume those features to produce boxes, masks, or labels. Getting the backbone right (both the architecture and the initialization), often matters more than the choice of head.

The Backbone's Role

A backbone is a deep CNN (or Vision Transformer) trained to extract hierarchically rich features from images. Detection and segmentation heads typically attach to intermediate feature maps at multiple scales (e.g., strides 8, 16, 32) rather than only the final layer, so that the head has access to both fine-grained detail and high-level semantics simultaneously.

Pre-training on ImageNet (1.28M images across 1,000 classes) provides a powerful weight initialization. Features learned for classifying cats, cars, and trees generalize broadly: early layers detect edges and textures; middle layers detect parts; late layers encode object-level semantics. Fine-tuning from this initialization converges faster and reaches higher accuracy than training from scratch — especially when downstream labeled data is limited.


ResNet: Residual Connections

Very deep networks (50+ layers) suffer from vanishing gradients: by the time the loss signal propagates back through dozens of layers, it shrinks to near zero and early layers stop learning.

Residual connections solve this by adding a shortcut path that bypasses one or more layers:

y=F(x,{Wi})+x\mathbf{y} = F(\mathbf{x}, \{W_i\}) + \mathbf{x}

where F(x)F(\mathbf{x}) is the residual mapping learned by the stacked convolutions. During backpropagation, gradients flow directly through the identity shortcut without passing through any weights — the gradient of y\mathbf{y} with respect to x\mathbf{x} has an additive term of 11 regardless of FF's weights.

Bottleneck Block

ResNet-50 and deeper variants use a three-layer bottleneck block to keep compute manageable:

  1. 1×11 \times 1 conv — reduce channels (e.g., 256→64256 \to 64)
  2. 3×33 \times 3 conv — spatial feature extraction at reduced channel count
  3. 1×11 \times 1 conv — expand channels back (e.g., 64→25664 \to 256)

The 3×33 \times 3 convolution operates on only 64 channels instead of 256, cutting compute by ≈4×\approx 4\times compared to a plain two-layer block.

ResNet Variants

ModelStages (blocks)ParamsGFLOPs
ResNet-503–4–6–325M4.1
ResNet-1013–4–23–344M7.8
ResNet-1523–8–36–360M11.6

ResNet-50 with a Feature Pyramid Network (FPN) head is the most common baseline backbone for detection and segmentation benchmarks.


MobileNet: Depthwise Separable Convolutions

A standard K×KK \times K convolution over CinC_{in} input channels producing CoutC_{out} output channels requires:

K×K×Cin×Cout multiplications per output locationK \times K \times C_{in} \times C_{out} \text{ multiplications per output location}

Depthwise separable convolution (introduced in depth in Modern Architectures) splits this into two steps:

  1. Depthwise conv: one K×KK \times K filter per input channel (no cross-channel mixing): K×K×CinK \times K \times C_{in} multiplications
  2. Pointwise conv: 1×1×Cin×Cout1 \times 1 \times C_{in} \times C_{out} (cross-channel mixing, no spatial): Cin×CoutC_{in} \times C_{out} multiplications

Cost reduction factor:

K2Cin+CinCoutK2CinCout=1Cout+1K2\frac{K^2 C_{in} + C_{in} C_{out}}{K^2 C_{in} C_{out}} = \frac{1}{C_{out}} + \frac{1}{K^2}

For K=3K = 3 and Cout=256C_{out} = 256: 1256+19≈0.115\frac{1}{256} + \frac{1}{9} \approx 0.115, meaning 8–9× cheaper than a standard convolution with minimal accuracy loss.

MobileNet Variants

MobileNetV2: Inverted Residuals. Counterintuitively, residual shortcuts connect narrow (low-channel) layers, while the 3×33 \times 3 depthwise conv operates in an expanded (high-channel) intermediate space. This prevents information loss in the bottleneck while keeping the shortcut cheap.

MobileNetV3: SE Attention + Hard-Swish. Adds Squeeze-and-Excite (SE) blocks that reweight channels by global context, and replaces swish activation with hard-swish (x⋅ReLU6(x+3)6\frac{x \cdot \text{ReLU6}(x+3)}{6}) — equivalent performance but integer-arithmetic friendly for mobile hardware.


EfficientNet: Compound Scaling

Prior approaches scaled networks along a single axis: make it deeper (more layers), wider (more channels), or feed higher-resolution inputs. EfficientNet's insight is that all three axes should be scaled jointly — depth, width, and resolution are interdependent.

Compound Scaling Rule

A single compound coefficient ϕ\phi controls total compute:

depth=1.2ϕ,width=1.1ϕ,resolution=1.15ϕ\text{depth} = 1.2^\phi, \quad \text{width} = 1.1^\phi, \quad \text{resolution} = 1.15^\phi

Total FLOPs scale as approximately 2ϕ2^\phi. The base ratios (1.2, 1.1, 1.15)(1.2,\ 1.1,\ 1.15) were found by neural architecture search (NAS) on a constrained grid.

EfficientNet-B0 to B7

Modelϕ\phiInputParamsTop-1
B002245.3M77.1%
B4438019M82.6%
B7760066M84.3%

EfficientNet-B0 was found by NAS to be the optimal baseline architecture for the given FLOP budget; B1–B7 scale it up using the compound rule. EfficientNet-B4 is a popular backbone choice when accuracy matters and compute is moderate.


Transfer Learning in Practice

Two Regimes

(The Transfer Learning lesson derives when each regime is safe from the trainable-params-vs-examples budget; here is the operational summary.)

Feature extraction — freeze all backbone weights; train only the task-specific head. Use when:

  • Downstream dataset is small (hundreds to low thousands of images)
  • Downstream task is similar to ImageNet classification

Full fine-tuning — unfreeze the entire network; train end-to-end with a low learning rate for the backbone. Use when:

  • Downstream dataset is large enough to support it
  • Downstream domain differs significantly from ImageNet (medical imaging, satellite imagery)

Layer-wise Learning Rate Decay

Earlier backbone layers learn general features (Gabor-like edges, color blobs) that generalize across tasks — they should change very little. Later layers encode task-specific semantics and can change more.

A common schedule: multiply the learning rate by a decay factor α<1\alpha < 1 for each earlier layer group. For example with α=0.1\alpha = 0.1 and a head LR of 10−310^{-3}:

  • Head: 10−310^{-3}
  • Final backbone stage: 2×10−42 \times 10^{-4}
  • Earlier stages: 10−410^{-4}, 2×10−52 \times 10^{-5}, …

This prevents destroying the general-purpose ImageNet features while still allowing fine-tuning for the new task.


ViT as Backbone

Vision Transformers (ViT) treat an image as a sequence of non-overlapping 16×1616 \times 16 patches. Each patch is flattened, projected to an embedding dimension, and processed by standard Transformer self-attention blocks.

  • ViT-B/16: 86M parameters, 17.6 GFLOPs per 224×224 image
  • Pre-trained on ImageNet-21k (14M images), then fine-tuned on ImageNet-1k, ViT-B/16 achieves 81.8% top-1 — competitive with ResNet-50 at 4× the compute
  • DeiT (Data-efficient Image Transformers) adds a distillation token to train ViT from scratch on ImageNet-1k without extra data, reaching 79.8% at ResNet-50 compute

ViT backbones excel when pre-training data is large. For smaller datasets or tighter compute budgets, ResNet and EfficientNet remain preferred.


Choosing a Backbone

BackboneParamsGFLOPsImageNet Top-1Use case
ResNet-5025M4.176%General baseline
MobileNetV3-L5.4M0.2275%Mobile / edge deployment
EfficientNet-B419M4.282.6%Accuracy/speed balance
ViT-B/1686M17.681.8%Large pre-training datasets

Rules of thumb:

  • Prototype or resource-constrained: MobileNetV3
  • Standard benchmark baseline: ResNet-50 + FPN
  • High-accuracy production model: EfficientNet-B4 or B5
  • Abundant compute and data: ViT-B/16 or larger

These accuracy columns compare training recipes as much as architectures. The ResNet-50 figure of 76% is the 2015 recipe: 90 epochs, step LR, basic augmentation. Re-train the identical architecture — same 25.6M parameters, same 4.1 GFLOPs — with a modern recipe (LAMB, heavy augmentation, longer schedule) and it reaches 80.4% top-1 (Wightman et al., ResNet strikes back, 2021; reproduced in timm's results table). That single change closes most of the gap to backbones released six years later.

Two consequences when you read any table like this one. First, a headline accuracy is a claim about a checkpoint, not an architecture — always check the recipe, the input resolution, and whether extra pre-training data was used (EfficientNet's highest published numbers use NoisyStudent with JFT-300M, which is not an ImageNet-only result). Second, FLOP counts follow two conventions that differ by 2×, since some authors report multiply-accumulates. Compare architectures only within a single consistent evaluation harness.

Always normalize inputs to ImageNet statistics when using any pre-trained backbone. The weights were trained expecting pixel values normalized to mean μ=[0.485, 0.456, 0.406]\mu = [0.485,\ 0.456,\ 0.406] and standard deviation σ=[0.229, 0.224, 0.225]\sigma = [0.229,\ 0.224,\ 0.225] (per channel, RGB). Skipping normalization produces activations outside the trained distribution and severely degrades transfer performance — even when fine-tuning end-to-end.


Exercise

A ResNet residual block computes F(x) + x, where F is two 3×3 conv layers with BN+ReLU. How do residual connections help train very deep networks (50+ layers)?

Need a hint?

Without skip connections, gradients must pass through every multiplication. What problem does this cause?

Exercise

A standard 3×3 convolution on C=256 input channels producing C=256 output channels has cost proportional to 3×3×256×256. A depthwise separable convolution splits this into depthwise (3×3×256) + pointwise (1×1×256×256). What is the approximate cost reduction?

Need a hint?

Compare the total multiply-add operations. Reduction ≈ 1/C_out + 1/K².

Exercise

EfficientNet scales depth by 1.2^φ, width by 1.1^φ, and resolution by 1.15^φ. At φ=2 (EfficientNet-B2), how does the input resolution compare to B0 (φ=0, baseline 224×224)?

Need a hint?

Resolution scale = 1.15^φ. Round to nearest multiple of 32 in practice.

Common Mistakes

Using ResNet-50 when MobileNet suffices. On simple classification tasks or edge deployment, ResNet-50 uses 19× more FLOPs than MobileNetV3-L for roughly the same accuracy. Always profile before defaulting to ResNet.

Fine-tuning with too high a learning rate. A learning rate suitable for training from scratch will destroy the learned ImageNet features in the first few gradient steps. Use 10−410^{-4} or lower for the backbone, with layer-wise decay toward early layers.

Not normalizing inputs to ImageNet mean/std. Pre-trained weights are tightly coupled to the statistics of their training data. Even after fine-tuning, starting with wrong normalization causes slow convergence and lower final accuracy.

Attaching heads only to the final backbone feature map. Heads that consume only stride-32 features lack the spatial resolution needed for small objects or fine segmentation boundaries. Use intermediate features (stride 8, 16, 32) via FPN or similar multi-scale structures.


Key Takeaways

  • Backbones extract hierarchical features from raw images; detection and segmentation heads attach to intermediate multi-scale feature maps.
  • ResNet's identity shortcut F(x)+xF(\mathbf{x}) + \mathbf{x} ensures gradient flow through very deep networks by adding an additive 11 term in the chain rule.
  • Depthwise separable convolutions reduce compute by a factor of 1/Cout+1/K21/C_{out} + 1/K^2 (roughly 8–9× for K=3K=3), enabling efficient mobile architectures.
  • EfficientNet scales depth, width, and resolution jointly with a compound coefficient ϕ\phi, achieving better accuracy per FLOP than single-axis scaling.
  • Transfer learning from ImageNet is almost always worth it; use feature extraction for small datasets and full fine-tuning (with low LR and layer-wise decay) for larger ones.
  • Always normalize inputs to ImageNet mean and standard deviation; never fine-tune a pre-trained backbone with the same learning rate as the head.

Related Concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Backbones in PracticeConvolutional Neural Netw…Attention & TransformersSegmentationModern CNN ArchitecturesTransfer LearningObject DetectionVision-Language ModelsNeural Radiance Fields (N…