Backbones in Practice
Every detection and segmentation model has two parts: a backbone that converts raw pixels into rich feature representations, and one or more heads that consume those features to produce boxes, masks, or labels. Getting the backbone right (both the architecture and the initialization), often matters more than the choice of head.
The Backbone's Role
A backbone is a deep CNN (or Vision Transformer) trained to extract hierarchically rich features from images. Detection and segmentation heads typically attach to intermediate feature maps at multiple scales (e.g., strides 8, 16, 32) rather than only the final layer, so that the head has access to both fine-grained detail and high-level semantics simultaneously.
Pre-training on ImageNet (1.28M images across 1,000 classes) provides a powerful weight initialization. Features learned for classifying cats, cars, and trees generalize broadly: early layers detect edges and textures; middle layers detect parts; late layers encode object-level semantics. Fine-tuning from this initialization converges faster and reaches higher accuracy than training from scratch — especially when downstream labeled data is limited.
ResNet: Residual Connections
Very deep networks (50+ layers) suffer from vanishing gradients: by the time the loss signal propagates back through dozens of layers, it shrinks to near zero and early layers stop learning.
Residual connections solve this by adding a shortcut path that bypasses one or more layers:
where is the residual mapping learned by the stacked convolutions. During backpropagation, gradients flow directly through the identity shortcut without passing through any weights — the gradient of with respect to has an additive term of regardless of 's weights.
Bottleneck Block
ResNet-50 and deeper variants use a three-layer bottleneck block to keep compute manageable:
- conv — reduce channels (e.g., )
- conv — spatial feature extraction at reduced channel count
- conv — expand channels back (e.g., )
The convolution operates on only 64 channels instead of 256, cutting compute by compared to a plain two-layer block.
ResNet Variants
| Model | Stages (blocks) | Params | GFLOPs |
|---|---|---|---|
| ResNet-50 | 3–4–6–3 | 25M | 4.1 |
| ResNet-101 | 3–4–23–3 | 44M | 7.8 |
| ResNet-152 | 3–8–36–3 | 60M | 11.6 |
ResNet-50 with a Feature Pyramid Network (FPN) head is the most common baseline backbone for detection and segmentation benchmarks.
MobileNet: Depthwise Separable Convolutions
A standard convolution over input channels producing output channels requires:
Depthwise separable convolution (introduced in depth in Modern Architectures) splits this into two steps:
- Depthwise conv: one filter per input channel (no cross-channel mixing): multiplications
- Pointwise conv: (cross-channel mixing, no spatial): multiplications
Cost reduction factor:
For and : , meaning 8–9× cheaper than a standard convolution with minimal accuracy loss.
MobileNet Variants
MobileNetV2: Inverted Residuals. Counterintuitively, residual shortcuts connect narrow (low-channel) layers, while the depthwise conv operates in an expanded (high-channel) intermediate space. This prevents information loss in the bottleneck while keeping the shortcut cheap.
MobileNetV3: SE Attention + Hard-Swish. Adds Squeeze-and-Excite (SE) blocks that reweight channels by global context, and replaces swish activation with hard-swish () — equivalent performance but integer-arithmetic friendly for mobile hardware.
EfficientNet: Compound Scaling
Prior approaches scaled networks along a single axis: make it deeper (more layers), wider (more channels), or feed higher-resolution inputs. EfficientNet's insight is that all three axes should be scaled jointly — depth, width, and resolution are interdependent.
Compound Scaling Rule
A single compound coefficient controls total compute:
Total FLOPs scale as approximately . The base ratios were found by neural architecture search (NAS) on a constrained grid.
EfficientNet-B0 to B7
| Model | Input | Params | Top-1 | |
|---|---|---|---|---|
| B0 | 0 | 224 | 5.3M | 77.1% |
| B4 | 4 | 380 | 19M | 82.6% |
| B7 | 7 | 600 | 66M | 84.3% |
EfficientNet-B0 was found by NAS to be the optimal baseline architecture for the given FLOP budget; B1–B7 scale it up using the compound rule. EfficientNet-B4 is a popular backbone choice when accuracy matters and compute is moderate.
Transfer Learning in Practice
Two Regimes
(The Transfer Learning lesson derives when each regime is safe from the trainable-params-vs-examples budget; here is the operational summary.)
Feature extraction — freeze all backbone weights; train only the task-specific head. Use when:
- Downstream dataset is small (hundreds to low thousands of images)
- Downstream task is similar to ImageNet classification
Full fine-tuning — unfreeze the entire network; train end-to-end with a low learning rate for the backbone. Use when:
- Downstream dataset is large enough to support it
- Downstream domain differs significantly from ImageNet (medical imaging, satellite imagery)
Layer-wise Learning Rate Decay
Earlier backbone layers learn general features (Gabor-like edges, color blobs) that generalize across tasks — they should change very little. Later layers encode task-specific semantics and can change more.
A common schedule: multiply the learning rate by a decay factor for each earlier layer group. For example with and a head LR of :
- Head:
- Final backbone stage:
- Earlier stages: , , …
This prevents destroying the general-purpose ImageNet features while still allowing fine-tuning for the new task.
ViT as Backbone
Vision Transformers (ViT) treat an image as a sequence of non-overlapping patches. Each patch is flattened, projected to an embedding dimension, and processed by standard Transformer self-attention blocks.
- ViT-B/16: 86M parameters, 17.6 GFLOPs per 224×224 image
- Pre-trained on ImageNet-21k (14M images), then fine-tuned on ImageNet-1k, ViT-B/16 achieves 81.8% top-1 — competitive with ResNet-50 at 4× the compute
- DeiT (Data-efficient Image Transformers) adds a distillation token to train ViT from scratch on ImageNet-1k without extra data, reaching 79.8% at ResNet-50 compute
ViT backbones excel when pre-training data is large. For smaller datasets or tighter compute budgets, ResNet and EfficientNet remain preferred.
Choosing a Backbone
| Backbone | Params | GFLOPs | ImageNet Top-1 | Use case |
|---|---|---|---|---|
| ResNet-50 | 25M | 4.1 | 76% | General baseline |
| MobileNetV3-L | 5.4M | 0.22 | 75% | Mobile / edge deployment |
| EfficientNet-B4 | 19M | 4.2 | 82.6% | Accuracy/speed balance |
| ViT-B/16 | 86M | 17.6 | 81.8% | Large pre-training datasets |
Rules of thumb:
- Prototype or resource-constrained: MobileNetV3
- Standard benchmark baseline: ResNet-50 + FPN
- High-accuracy production model: EfficientNet-B4 or B5
- Abundant compute and data: ViT-B/16 or larger
These accuracy columns compare training recipes as much as architectures. The ResNet-50 figure of 76% is the 2015 recipe: 90 epochs, step LR, basic augmentation. Re-train the identical architecture — same 25.6M parameters, same 4.1 GFLOPs — with a modern recipe (LAMB, heavy augmentation, longer schedule) and it reaches 80.4% top-1 (Wightman et al., ResNet strikes back, 2021; reproduced in timm's results table). That single change closes most of the gap to backbones released six years later.
Two consequences when you read any table like this one. First, a headline accuracy is a claim about a checkpoint, not an architecture — always check the recipe, the input resolution, and whether extra pre-training data was used (EfficientNet's highest published numbers use NoisyStudent with JFT-300M, which is not an ImageNet-only result). Second, FLOP counts follow two conventions that differ by 2×, since some authors report multiply-accumulates. Compare architectures only within a single consistent evaluation harness.
Always normalize inputs to ImageNet statistics when using any pre-trained backbone. The weights were trained expecting pixel values normalized to mean and standard deviation (per channel, RGB). Skipping normalization produces activations outside the trained distribution and severely degrades transfer performance — even when fine-tuning end-to-end.
Exercise
A ResNet residual block computes F(x) + x, where F is two 3×3 conv layers with BN+ReLU. How do residual connections help train very deep networks (50+ layers)?
Need a hint?
Without skip connections, gradients must pass through every multiplication. What problem does this cause?
Exercise
A standard 3×3 convolution on C=256 input channels producing C=256 output channels has cost proportional to 3×3×256×256. A depthwise separable convolution splits this into depthwise (3×3×256) + pointwise (1×1×256×256). What is the approximate cost reduction?
Need a hint?
Compare the total multiply-add operations. Reduction ≈ 1/C_out + 1/K².
Exercise
EfficientNet scales depth by 1.2^φ, width by 1.1^φ, and resolution by 1.15^φ. At φ=2 (EfficientNet-B2), how does the input resolution compare to B0 (φ=0, baseline 224×224)?
Need a hint?
Resolution scale = 1.15^φ. Round to nearest multiple of 32 in practice.
Common Mistakes
Using ResNet-50 when MobileNet suffices. On simple classification tasks or edge deployment, ResNet-50 uses 19× more FLOPs than MobileNetV3-L for roughly the same accuracy. Always profile before defaulting to ResNet.
Fine-tuning with too high a learning rate. A learning rate suitable for training from scratch will destroy the learned ImageNet features in the first few gradient steps. Use or lower for the backbone, with layer-wise decay toward early layers.
Not normalizing inputs to ImageNet mean/std. Pre-trained weights are tightly coupled to the statistics of their training data. Even after fine-tuning, starting with wrong normalization causes slow convergence and lower final accuracy.
Attaching heads only to the final backbone feature map. Heads that consume only stride-32 features lack the spatial resolution needed for small objects or fine segmentation boundaries. Use intermediate features (stride 8, 16, 32) via FPN or similar multi-scale structures.
Key Takeaways
- Backbones extract hierarchical features from raw images; detection and segmentation heads attach to intermediate multi-scale feature maps.
- ResNet's identity shortcut ensures gradient flow through very deep networks by adding an additive term in the chain rule.
- Depthwise separable convolutions reduce compute by a factor of (roughly 8–9× for ), enabling efficient mobile architectures.
- EfficientNet scales depth, width, and resolution jointly with a compound coefficient , achieving better accuracy per FLOP than single-axis scaling.
- Transfer learning from ImageNet is almost always worth it; use feature extraction for small datasets and full fine-tuning (with low LR and layer-wise decay) for larger ones.
- Always normalize inputs to ImageNet mean and standard deviation; never fine-tune a pre-trained backbone with the same learning rate as the head.
Related Concepts
- Transfer Learning: fine-tuning strategies and when to freeze vs. unfreeze layers
- Object Detection: FPN and multi-scale backbone feature usage in detection heads
- Segmentation: U-Net and Mask R-CNN encoder backbones
- Neural Radiance Fields (NeRF): coordinate-based neural fields that extend representation learning to 3D scenes