Object Detection
A classifier answers "what is in this image?". An object detector answers "what is in this image, where exactly is each one, and how confident are you?" That additional where transforms a single label into a list of bounding boxes, each paired with a class and a confidence score. Most of this course lives in the hypothesis-space slot of the ML project loop (anchors, FPNs, U-Nets, and backbones are all architectural machinery) with one big exception: self-supervised learning (lesson 05), which is squarely about the data slot.
The Detection Problem
Three tasks live on a spectrum of spatial precision:
- Image classification: one label for the whole image (e.g. "cat").
- Object localisation: one label plus one bounding box (single dominant object).
- Object detection: multiple labels, each with its own bounding box and confidence score, for every object instance in the scene.
A bounding box is parameterised as , where is the box centre and are its width and height, usually normalised to the image dimensions so values fall in .
The detector output for a single image is a variable-length list of tuples:
where is the predicted class index and is the confidence score.
Anchor Boxes
Modern detectors do not predict box coordinates from scratch. Instead they predict offsets from a set of anchor boxes — predefined rectangles of varying scale and aspect ratio tiled across the image grid.
Why anchors? Raw coordinate regression is unbounded and unstable to train. Predicting small offsets from a well-chosen prior is a far easier learning problem.
Typical anchor design: At each grid cell location, place 3 scales × 3 aspect ratios = 9 anchor boxes. Common aspect ratios are ; common scales are pixels (relative to the input image).
For a backbone with output stride 32 on a image, the feature map is , giving anchors in total.
Intersection over Union (IoU)
To decide whether an anchor matches a ground-truth box, we measure their spatial overlap with IoU:
is the area of the rectangle formed by the intersection of boxes and ; .
IoU ranges from 0 (no overlap) to 1 (perfect overlap). The standard assignment rule is:
- IoU with any ground-truth box → positive anchor (foreground, train box regression and class head)
- IoU → negative anchor (background, train class head only)
- IoU → ignored during training
The model predicts four offsets that transform the anchor into the predicted box via:
(and symmetrically for ). The exponential ensures width/height stay positive.
Counting what that rule actually labels
Seven anchors out of three thousand
A 16×16 grid at 4 scales and 3 aspect ratios — 3072 anchors over a 512×512 image. Teal boxes are the anchors labelled positive at the current threshold; the white box is the ground truth and the yellow one is the single best-matching anchor. Every count is computed over all 3072.
Run the assignment rule over all 3072 anchors against a single object and the numbers are stark: 7 positive, 12 ignored, 3053 negative — a foreground-to-background ratio of 1:436. That single count is the reason detection has a whole sub-literature on class imbalance. Focal loss, hard-negative mining, and the 1:3 sampling ratio in Faster R-CNN all exist to stop those 3053 easy negatives from drowning the 7 examples that carry the signal.
Two things are worth pushing on.
Raise the positive threshold to 0.7. It does not produce cleaner labels; it produces no labels. The best IoU any anchor achieves on this box is , so a stricter rule leaves the object with zero positives and it drops out of training entirely. This is exactly why the convention sits at , and why real detectors add a best-anchor-per-object fallback on top of the IoU test — without it, an unlucky object is simply never learned.
Switch shapes. Best achievable IoU by object geometry, on this anchor set:
| object | best IoU available |
|---|---|
| typical, 140×180 | 0.635 |
| wide, 320×80 | 0.527 |
| thin, 60×400 | 0.368 |
| small, 32×32 | 0.250 |
The last two never clear any sensible positive threshold, so they are invisible to the detector regardless of how long you train. Anchor scales and aspect ratios are a decision about your dataset, not a default to inherit, and this is the failure mode that anchor-free detectors (FCOS, CenterNet) were designed to remove.
Faster R-CNN: Two-Stage Detection
Faster R-CNN (Ren et al., 2015) is the canonical two-stage detector: first propose regions of interest, then classify them.
Stage 1: Region Proposal Network (RPN)
The RPN is a small convolutional head attached to the backbone. It slides over every location in the backbone feature map and, for each anchor:
- Predicts an objectness score (is there any object here?).
- Predicts box offsets to refine the anchor into a region proposal.
The RPN outputs ~2 000 candidate regions after filtering by objectness and applying NMS (see below).
Stage 2: RoI Pooling + Classification Head
Each proposal is cropped from the backbone feature map using RoI Pooling (or the smoother RoIAlign variant), which resamples the variable-size proposal into a fixed feature grid. This fixed-size feature is then fed to:
- A classification head predicting one of classes (plus background).
- A regression head predicting final box offsets per class.
Speed vs accuracy: Faster R-CNN runs at roughly 5–10 fps on a GPU (too slow for real-time use) but delivers high accuracy because the two-stage design allows the classification head to focus only on promising regions.
Training loss:
is cross-entropy; is smooth- (Huber) loss on box offsets. balances the two terms (typically ).
Objectness vs class confidence in YOLO
YOLO predicts two distinct scores per box: an objectness score — does any object exist here at all? — and a set of class probabilities . The final detection confidence is their product: . Faster R-CNN keeps these roles separated between the RPN (objectness) and the classification head (class probabilities). Conflating the two is a common source of confusion when reading detection papers.
YOLO: One-Stage Detection
You Only Look Once (Redmon et al., 2016) collapses the two stages into a single forward pass, making it fast enough for real-time applications.
Grid-Based Prediction
YOLO divides the input image into an grid. Each cell is responsible for detecting objects whose centres fall within it. Each cell predicts bounding boxes and class probabilities simultaneously.
The output is a single tensor of shape:
For YOLOv1 with , , (PASCAL VOC):
Each of the entries encodes for one box.
Multi-Scale Predictions (YOLOv3+)
YOLOv1 and v2 struggle with small and overlapping objects because a single grid is too coarse. YOLOv3 addresses this by predicting at three scales (analogous to an FPN) using feature maps of strides 8, 16, and 32. Larger feature maps (stride 8) detect small objects; smaller maps (stride 32) detect large objects.
Speed: YOLOv1 achieves ~45 fps; modern variants (YOLOv8, RT-DETR) push well past 60 fps at competitive accuracy.
Feature Pyramid Networks (FPN)
A fundamental challenge in detection is scale invariance: a stop sign close to the camera occupies thousands of pixels; the same sign far away occupies dozens. A single feature map at one resolution cannot handle both cases well.
FPN (Lin et al., 2017) solves this with a two-pathway architecture:
- Bottom-up pathway: the standard backbone forward pass, producing feature maps at strides 4, 8, 16, 32 (denoted C2–C5).
- Top-down pathway: start from the deepest (most semantic) feature map C5 and upsample 2× at each level, adding lateral connections from the corresponding bottom-up map via convolutions.
The result is a pyramid of feature maps P2–P5, each with 256 channels, all enriched with both low-level spatial detail and high-level semantic context. Detection heads run independently at each scale.
FPN is now a standard component in Faster R-CNN, Mask R-CNN, RetinaNet, and many other architectures.
Non-Maximum Suppression (NMS)
After a forward pass, a detector with thousands of anchors will produce thousands of overlapping boxes for the same object. NMS prunes duplicates by iteratively keeping the highest-confidence box and suppressing all boxes whose IoU with it exceeds a threshold (typically 0.5). For the full 6-step algorithm, the 4-box worked trace (confidences 0.92, 0.87, 0.61, 0.55), and the Soft-NMS Gaussian decay variant for crowded scenes:
SSD: Single Shot MultiBox Detector
SSD (Liu et al., 2016) is a one-stage detector that, like YOLO, avoids explicit region proposals. Unlike YOLO's single output tensor, SSD attaches detection heads directly to multiple backbone feature maps at different resolutions (e.g. 38×38, 19×19, 10×10, 5×5, 3×3, 1×1), placing default boxes (anchors) of 6 different aspect ratios at each location.
This multi-scale strategy is conceptually similar to FPN but uses the raw backbone maps rather than a dedicated top-down pathway. SSD is fast and straightforward to implement; it is less accurate than FPN-based models on small objects because the shallow early feature maps lack semantic richness.
Two-Stage vs One-Stage: Comparison
| Property | Faster R-CNN | YOLO / SSD |
|---|---|---|
| Speed | 5–10 fps | 30–60+ fps |
| Accuracy (COCO mAP) | Higher | Lower (gap closing) |
| Proposal mechanism | RPN (explicit) | Grid cells (implicit) |
| Small objects | Good | Struggles (YOLO v1/v2) |
| Typical use case | Offline, accuracy-critical | Real-time applications |
The accuracy gap has narrowed considerably with YOLOv7/v8 and one-stage detectors with FPN backbones. For latency-sensitive applications (autonomous vehicles, robotics, live video), one-stage detectors remain the dominant choice.
Evaluation: Mean Average Precision (mAP)
A single accuracy number is not enough for detection, because you must reward both correct classification and correct localisation.
Precision-Recall curve: For a given class, vary the confidence threshold. At each threshold, compute:
A predicted box counts as a true positive (TP) only if its IoU with a ground-truth box of the same class exceeds 0.5, and that ground-truth box has not already been matched.
Average Precision (AP): the area under the precision-recall curve for one class. AP uses IoU threshold 0.5.
mAP: mean of AP across all classes. COCO mAP (written mAP or AP) averages AP over IoU thresholds 0.50, 0.55, …, 0.95 (ten thresholds), weighting precise localisation much more heavily than the simpler VOC metric.
Exercise
An anchor box has area 100×100 pixels and a ground-truth box has area 80×80 pixels. They overlap with an intersection area of 4000 pixels². What is the IoU, and is this anchor positive (IoU > 0.5)?
Need a hint?
IoU = intersection / (A + B − intersection). Union = area_A + area_B − intersection.
Exercise
Non-Maximum Suppression (NMS) is applied to 5 detected boxes for 'cat', sorted by confidence: [0.95, 0.88, 0.72, 0.61, 0.45]. The IoU threshold is 0.5. Box 1 (0.95) has IoU > 0.5 with boxes 2 and 3. What does NMS output?
Need a hint?
NMS keeps the highest-confidence box, removes overlapping boxes, then repeats on the remaining set.
Exercise
YOLOv1 divides a 448×448 image into a 7×7 grid. Each cell predicts B=2 bounding boxes and C=20 class probabilities. What is the shape of the output tensor?
Need a hint?
Each cell outputs: B boxes × 5 values (x, y, w, h, confidence) + C class probabilities.
Common Mistakes and Pitfalls
- Anchors too small for large objects: always visualise your anchor coverage on a few training images before training. Under-coverage of large objects leads to many missed detections.
- Forgetting to normalise bounding box coordinates: some frameworks expect in pixels; others expect values in . Mixing conventions causes boxes to appear wildly wrong.
- NMS threshold too aggressive (0.7+) or too lenient (0.3-): at 0.7, heavily overlapping duplicates survive; at 0.3, nearby distinct objects are suppressed. 0.45–0.5 is the standard starting point.
- Not using multi-scale training: randomly resizing inputs during training (e.g. between 320 and 608 pixels) dramatically improves robustness to scale at test time.
- Ignoring class imbalance in anchors: in a typical image, 99%+ of anchors are background. Without focal loss (RetinaNet) or hard negative mining (SSD/Faster R-CNN), the background overwhelms the training signal.
Key Takeaways
- Object detection outputs a variable-length list of tuples — one per detected object instance.
- Anchor boxes are predefined priors; models predict small offsets from them rather than absolute coordinates, which makes regression stable.
- IoU measures bounding-box overlap; the standard positive-anchor threshold is IoU .
- Faster R-CNN (two-stage) is accurate but slow; YOLO/SSD (one-stage) are fast but historically less accurate on small objects.
- FPN adds a top-down lateral pathway to backbone features, giving every scale access to high-level semantics. This is the key ingredient for detecting objects across a wide size range.
- NMS removes duplicate detections by keeping the highest-confidence box and suppressing all boxes with IoU threshold against it; Soft-NMS is preferred in crowded scenes.
Related Concepts
- Convolution Operation: the building block of all detection backbones.
- Segmentation: the next step: predicting a class label for every pixel, not just a bounding box.