Transfer Learning
Training a CNN from scratch requires millions of images and days of computation. Transfer learning lets you reuse a pretrained network and adapt it to your task with much less data.
Real-world examples: Medical imaging (only hundreds of X-rays available), satellite imagery classification, industrial defect detection, rare species identification.
The key insight
A CNN trained on ImageNet (1.2M images, 1000 classes) has learned:
- Early layers: universal features — edges, textures, colors
- Middle layers: object parts — eyes, wheels, leaves
- Late layers: task-specific features — dog breeds, car models
The early and middle layers are universal — they work for almost any image task. Only the late layers need adaptation.
Intuition
It's like learning to paint. First you learn to see shapes, colors, and light (universal skills). Then you specialize — portraits, landscapes, abstract art. Transfer learning skips the "learning to see" part.
Why freezing early layers works: the gradient view
"Freezing" a layer means setting requires_grad = False on its parameters. Concretely, during backprop the optimizer simply skips the weight update for that layer:
This is justified by two facts:
-
Early-layer features are already near-optimal. Gabor-like edge and color detectors emerge in the first conv layers of almost every vision model regardless of the final task. Their gradient on your small dataset is mostly noise — updating them risks overfitting more than it helps.
-
Frozen layers cost nothing to "train." Their gradients are never computed, so each step is cheaper, and the optimizer has far fewer parameters to estimate — exactly what you want when data is scarce. With trainable parameters you need roughly examples to constrain them; freezing slashes by orders of magnitude (we count this exactly below).
The forward pass still flows through the frozen layers — they act as a fixed feature extractor turning your image into a 2048-dimensional embedding (for ResNet-50). Only the head learns to map that embedding to your classes.
Three strategies
Strategy 1: Feature extraction (least data)
Freeze all pretrained layers, replace only the final classifier:
import torch.nn as nn
from torchvision import models
# Load pretrained ResNet
model = models.resnet50(weights='IMAGENET1K_V2')
# Freeze all layers
for param in model.parameters():
param.requires_grad = False
# Replace classifier (only this part trains)
model.fc = nn.Sequential(
nn.Linear(2048, 256),
nn.ReLU(),
nn.Dropout(0.3),
nn.Linear(256, num_classes)
)
# Count what's actually trainable
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"Trainable: {trainable:,} / {total:,} ({100*trainable/total:.1f}%)")
When to use: < 1,000 training images. The pretrained features are already good — you just need a simple classifier.
Worked parameter count: swapping the head on ResNet-50
ResNet-50 ships with 25,557,032 parameters and an ImageNet head fc = Linear(2048, 1000). A fully-connected layer with inputs and outputs has
Original head ():
Strip it off and the frozen backbone keeps the rest:
Now bolt on the new two-layer head above for, say, classes (pneumonia vs normal):
So the model now has parameters, but only the head is trainable:
Only 2.2% of the network learns. If you instead used a plain Linear(2048, 2) head ( params) the trainable fraction drops to a stunning 0.017%. You are fitting ~4K numbers, which a few hundred images can comfortably constrain. This is why feature extraction works on tiny datasets.
Strategy 2: Fine-tuning (medium data)
Unfreeze the last few layers and train with a small learning rate:
# Unfreeze last 2 blocks
for param in model.layer4.parameters():
param.requires_grad = True
# Use different learning rates
optimizer = torch.optim.Adam([
{'params': model.layer4.parameters(), 'lr': 1e-4}, # small LR
{'params': model.fc.parameters(), 'lr': 1e-3} # normal LR
])
When to use: 1,000-100,000 images. The task is somewhat different from ImageNet (e.g., medical images vs natural images).
"Last few layers" is not a few parameters
It's tempting to think unfreezing layer4 only adds a little. In ResNet-50, layer4 is the widest stage (it works with up to 2048 channels), so it alone holds about 14.96M parameters — roughly 58.6% of the whole network:
Unfreezing layer4 plus the new head from Strategy 1 makes
parameters trainable — about 64% of the model. So "fine-tune the last block" is really "retrain the majority of the weights." That is exactly why you need more data (1K–100K) and a smaller learning rate than feature extraction.
Why discriminative (layer-wise) learning rates
The two parameter groups above use different learning rates on purpose:
- The new head is randomly initialized — its weights are garbage, so it needs a normal-sized LR () to learn fast.
- The pretrained
layer4is already excellent — it only needs gentle nudging, so it gets a smaller LR ().
If you used the head's large LR on the backbone, the first few mini-batches (when the head's gradients are huge and meaningless) would flow back and overwrite the pretrained features before the head has learned anything useful. This is catastrophic forgetting. Discriminative LRs (often a geometric decay like for layer , with –) keep early layers nearly fixed while letting the head move freely.
A common companion trick is to warm up the head first: freeze the backbone for a few epochs (feature extraction) until the head's gradients are sane, then unfreeze layer4 for fine-tuning.
Strategy 3: Full fine-tuning (lots of data)
Unfreeze everything and train with a very small learning rate:
# Unfreeze everything
for param in model.parameters():
param.requires_grad = True
# Very small learning rate to avoid destroying pretrained features
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)
When to use: > 100,000 images. The task is very different from ImageNet (e.g., satellite imagery, medical scans).
Transfer learning: freeze / unfreeze depth
Frozen layers (gray) keep ImageNet weights intact; trainable layers (purple) update during fine-tuning. Use the slider or preset buttons to explore the three transfer strategies.
Decision guide
The choice hinges on two axes: how much labeled data you have (more data lets you safely train more parameters without overfitting) and how different your images are from ImageNet (a bigger domain gap means the late, task-specific features transfer poorly and must be relearned).
| Data Size | Task Similarity | Strategy |
|---|---|---|
| < 1K | Similar to ImageNet | Feature extraction |
| < 1K | Different from ImageNet | Feature extraction + augmentation |
| 1K-100K | Similar | Fine-tune last layers |
| 1K-100K | Different | Fine-tune more layers |
| > 100K | Any | Full fine-tuning |
The logic of the table is a simple guardrail: trainable parameters examples. With ~1K images you can responsibly fit the ~4K–525K parameters of a new head (feature extraction), but not the ~15M of layer4 — you'd overfit. As data grows you can afford to unfreeze more, and a large domain gap forces you to (because the frozen features no longer match your images).
Learning rate is critical
When fine-tuning, use a learning rate 10-100× smaller than training from scratch. The pretrained weights sit in a good local minimum; a from-scratch LR produces large updates whose first steps move the weights out of that minimum faster than the loss can pull them back — you destroy in one epoch what ImageNet spent days learning. Pair the small LR with a scheduler (cosine or step decay) so updates shrink further as you approach the new optimum.
Practical tips
-
Start with feature extraction: It's fast and often sufficient. Only move to fine-tuning if accuracy is insufficient.
-
Use data augmentation — Transfer learning shines with small datasets. Augment aggressively:
from torchvision import transforms
transform = transforms.Compose([
transforms.RandomHorizontalFlip(),
transforms.RandomRotation(15),
transforms.ColorJitter(brightness=0.2, contrast=0.2),
transforms.RandomAffine(degrees=0, translate=(0.1, 0.1)),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])
-
Use the right pretrained model:
- ResNet-50: Good default, balanced speed/accuracy
- EfficientNet-B0: Best accuracy per parameter
- MobileNet: Mobile/edge deployment
- ViT: State-of-the-art but needs more data
-
Monitor overfitting: With small datasets, overfitting is the #1 enemy. Use early stopping, dropout, and weight decay.
Worked example: Medical image classification
Task: Classify X-rays as pneumonia vs normal (1,000 training images).
import torch
from torchvision import models, transforms
from torch.utils.data import DataLoader
# 1. Load pretrained model
model = models.resnet50(weights='IMAGENET1K_V2')
# 2. Freeze early layers
for param in model.parameters():
param.requires_grad = False
# 3. Replace classifier with a 2-class head
model.fc = nn.Linear(2048, 2) # 2048*2 + 2 = 4,098 trainable params
# 4. Train with appropriate settings
optimizer = torch.optim.Adam(model.fc.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()
# Of ResNet-50's ~25.5M params, only 4,098 (0.017%) update here.
# With only 1K images, feature extraction typically achieves 85-90% accuracy
# Fine-tuning the last block might push it to 90-93%
# Training from scratch with 1K images: likely < 70%
With 1,000 images and a 4,098-parameter head you are fitting only ~4 parameters per training image — comfortably constrained. Training all 25.5M parameters from scratch on the same data means ~25,557 parameters per image, vastly underdetermined, which is why it overfits to near-uselessness on this dataset.
Exercise
You have only 200 medical images. What's the best transfer learning strategy?
Need a hint?
Think about how much data you need to fine-tune vs freeze.
Exercise
When fine-tuning a pretrained CNN, what learning rate should you use?
Need a hint?
The pretrained weights are already good — you don't want to destroy them.
Exercise
In transfer learning, which layers are most task-specific and should be fine-tuned first?
Need a hint?
Think about what early vs late layers detect.
Exercise
Set the dropout rate for a fully-connected layer in a CNN fine-tuned on a small dataset (~1 000 images).
Need a hint?
Too low = no regularization; too high = most activations drop and the network can't learn.
Common mistakes
- Using ImageNet normalization on non-RGB data: Medical images, grayscale, or satellite images need different normalization stats.
- Fine-tuning too aggressively: Large learning rates destroy pretrained features. Start small.
- Ignoring domain mismatch: If your images look very different from ImageNet (e.g., X-rays, aerial photos), the pretrained features may not transfer well. Consider training from scratch if you have enough data.
Key takeaways
- Transfer learning reuses features learned on a huge dataset (ImageNet) for your task.
- Freezing a layer discards its gradient so its weights never update — the backbone becomes a fixed feature extractor and only the head learns.
- A swapped head is tiny: ResNet-50's ImageNet head ( params) becomes a head ( params), and feature extraction trains as little as 0.017% of the network.
- "Fine-tune the last block" is misleadingly small: ResNet-50's
layer4is ~15M params (58.6% of the model), so unfreezing it retrains the majority of the weights — hence the need for more data and a smaller LR. - Use discriminative learning rates (small for the pretrained backbone, larger for the new head) and a from-scratch-÷10–100 LR to avoid catastrophic forgetting.
- Little data → freeze the backbone, train a new head; lots → fine-tune more layers.
Related concepts
- Pooling & Architectures: the pretrained backbones you transfer
- Gradient Descent: why fine-tuning needs a small learning rate