🎯 What You'll Learn
- Understand the four levels of computer vision understanding: classification, detection, semantic segmentation, and instance segmentation
- Calculate IoU (Intersection over Union) and implement Non-Maximum Suppression (NMS) from scratch
- Understand the R-CNN family: R-CNN → Fast R-CNN → Faster R-CNN with Region Proposal Networks
- Use YOLO one-stage detectors for real-time detection with the Ultralytics library
- Understand Feature Pyramid Networks (FPN) for multi-scale detection
- Build a U-Net architecture for semantic segmentation with skip connections
- Use pre-trained detection models from torchvision and Hugging Face, evaluate with mAP
Image classification says "this image contains a cat." Object detection says "there is a cat at (x=120, y=80, w=200, h=180) and a dog at (x=350, y=100, w=180, h=160)." Semantic segmentation says "every pixel that belongs to a cat is labeled 'cat', every pixel of road is labeled 'road'." As you move from classification → detection → segmentation, you gain more spatial precision but need more powerful architectures. These are the core tasks powering self-driving cars, medical imaging, and AR/VR.
1 The Three Levels of Computer Vision Understanding
Computer vision has evolved through progressively richer levels of spatial understanding. Each level answers a more detailed question about what is in an image and where.
Level 1: Image Classification
The simplest task: assign one label to the entire image. A ResNet given a photo of a cat outputs "cat" with 97% confidence. There is no spatial information — you just know something is there. This is sufficient for filtering photos (is this a landscape or a portrait?) but useless if you need to know where objects are located.
Level 2: Object Detection
A bounding box (a rectangle) plus a class label for every detected object instance in the image. Given a street scene, a detector might output: {class: "car", box: [120, 80, 320, 260], confidence: 0.95}, {class: "pedestrian", box: [450, 90, 510, 280], confidence: 0.88}. This is what powers face detection in cameras, pedestrian detection in cars, and product recognition in retail apps.
Level 3: Semantic Segmentation
Every single pixel in the image gets a class label. A semantic segmentation model on a street scene labels every pixel as one of: road, sidewalk, car, building, sky, pedestrian, tree. Crucially, if there are two cars, all their pixels share the same "car" label — there is no distinction between individual instances. This is ideal when you care about regions (how much of this image is sky?) rather than individual objects.
Level 4: Instance Segmentation
Combines detection and segmentation. Each detected object instance gets its own pixel-level mask. Two cars get two separate masks. A person partially behind a tree gets a mask for only the visible pixels. This is what Mask R-CNN produces — both a bounding box and a per-pixel mask for each detected object. It is the most spatially precise and also the most computationally expensive.
Bonus: Panoptic Segmentation
The most complete form: every pixel is labeled, with "things" (countable objects like cars, people) getting instance-level labels and "stuff" (uncountable regions like sky, grass, road) getting semantic labels. This is what self-driving car perception systems use internally.
The distinction between semantic, instance, and panoptic segmentation trips up almost everyone the first time — the three diagrams below show the same scene (two overlapping cars in front of a road) labeled three different ways:
Same scene, three labeling schemes. Semantic segmentation merges both cars into one "car" class (left). Instance segmentation separates car #1 from car #2 but doesn't bother labeling background "stuff" like road (middle). Panoptic segmentation does both at once — semantic labels for stuff, instance labels for things (right).
| Task | Output | Distinguishes Instances? | Use Case |
|---|---|---|---|
| Classification | 1 class label per image | No | Photo sorting, content filtering |
| Object Detection | Bounding boxes + class labels | Yes (by box) | Surveillance, autonomous driving, retail |
| Semantic Seg. | Per-pixel class label | No | Road scene understanding, medical imaging |
| Instance Seg. | Per-pixel mask per instance | Yes (by mask) | Robot manipulation, precise counting |
| Panoptic Seg. | Semantic + instance combined | Yes (full) | Autonomous driving, full scene understanding |
2 Bounding Boxes and IoU
Before studying detectors, you need a firm grip on the geometry. Two concepts underpin everything: bounding box formats and Intersection over Union (IoU).
Bounding Box Formats
There are two common formats. PASCAL VOC format uses corner coordinates: (x_min, y_min, x_max, y_max) — the top-left corner and bottom-right corner. YOLO format uses center coordinates: (x_center, y_center, width, height) — often normalized to [0, 1] by dividing by image dimensions. You'll encounter both in different datasets and libraries — always verify which format your code expects.
def convert_box_format(box, from_fmt='xyxy', to_fmt='xywh'):
"""Convert between (x_min,y_min,x_max,y_max) and (x_center,y_center,w,h)."""
if from_fmt == 'xyxy' and to_fmt == 'xywh':
x_min, y_min, x_max, y_max = box
w = x_max - x_min
h = y_max - y_min
x_c = x_min + w / 2
y_c = y_min + h / 2
return [x_c, y_c, w, h]
elif from_fmt == 'xywh' and to_fmt == 'xyxy':
x_c, y_c, w, h = box
return [x_c - w/2, y_c - h/2, x_c + w/2, y_c + h/2]
# Example: a box from (100,150) to (300,400)
box_xyxy = [100, 150, 300, 400]
box_xywh = convert_box_format(box_xyxy, 'xyxy', 'xywh')
print(f"Corner format: {box_xyxy}") # [100, 150, 300, 400]
print(f"Center format: {box_xywh}") # [200.0, 275.0, 200, 250]
IoU: Intersection over Union
IoU measures how well a predicted box overlaps with the ground truth box. It is the single most important metric in object detection. IoU = Area(Prediction ∩ Ground Truth) / Area(Prediction ∪ Ground Truth). A perfect prediction has IoU = 1.0. Zero overlap has IoU = 0.0. The standard threshold for calling a detection "correct" is IoU ≥ 0.5, though COCO evaluation uses thresholds from 0.5 to 0.95 in 0.05 steps.
def compute_iou(box1, box2):
"""
Compute IoU between two boxes in (x_min, y_min, x_max, y_max) format.
Returns a float in [0, 1].
"""
x1 = max(box1[0], box2[0])
y1 = max(box1[1], box2[1])
x2 = min(box1[2], box2[2])
y2 = min(box1[3], box2[3])
# Intersection area (clamp to 0 if boxes don't overlap)
inter_area = max(0, x2 - x1) * max(0, y2 - y1)
# Areas of each box
area1 = (box1[2] - box1[0]) * (box1[3] - box1[1])
area2 = (box2[2] - box2[0]) * (box2[3] - box2[1])
# Union = sum of areas minus intersection
union_area = area1 + area2 - inter_area
if union_area == 0:
return 0.0
return inter_area / union_area
# Example 1: Perfect overlap
box_pred = [100, 100, 300, 300]
box_gt = [100, 100, 300, 300]
print(f"IoU (perfect): {compute_iou(box_pred, box_gt):.3f}") # 1.000
# Example 2: 50% overlap
box_pred = [100, 100, 300, 300]
box_gt = [200, 100, 400, 300]
print(f"IoU (partial): {compute_iou(box_pred, box_gt):.3f}") # 0.333
# Example 3: No overlap
box_pred = [100, 100, 200, 200]
box_gt = [300, 300, 400, 400]
print(f"IoU (no overlap): {compute_iou(box_pred, box_gt):.3f}") # 0.000
Build Your Intuition: Drag the Predicted Box
The ground-truth box (green, fixed) sits in the middle of the canvas. Use the sliders to slide the predicted box (amber) horizontally and vertically, and to resize its width and height. Watch the intersection region (shaded) shrink and grow, and watch the live IoU score track it — including the moment it crosses the IoU ≥ 0.5 line that most benchmarks use to call a detection "correct."
IoU = 0.000
Non-Maximum Suppression (NMS)
Object detectors produce many overlapping boxes for the same object. For a single car, the model might predict 12 slightly different boxes, all with high confidence. NMS is the post-processing step that removes duplicates: keep the highest-confidence box, then suppress all boxes with IoU greater than a threshold (typically 0.5) with the kept box. Repeat until no boxes remain to suppress.
def nms(boxes, scores, iou_threshold=0.5):
"""
Non-Maximum Suppression.
boxes: list of [x_min, y_min, x_max, y_max]
scores: confidence score for each box
Returns: indices of kept boxes
"""
# Sort boxes by score (highest first)
order = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)
keep = []
while order:
# Always keep the highest-confidence remaining box
i = order[0]
keep.append(i)
remaining = []
# Remove boxes with high IoU with box i
for j in order[1:]:
if compute_iou(boxes[i], boxes[j]) < iou_threshold:
remaining.append(j) # keep low-overlap boxes
order = remaining # continue with surviving boxes
return keep
# Example: 5 boxes predicting the same car
boxes = [
[100, 100, 300, 300], # idx 0: score 0.95
[105, 102, 305, 298], # idx 1: score 0.88 (overlaps 0)
[98, 99, 295, 302], # idx 2: score 0.80 (overlaps 0)
[400, 400, 550, 550], # idx 3: score 0.72 (different object)
[102, 103, 302, 299], # idx 4: score 0.60 (overlaps 0)
]
scores = [0.95, 0.88, 0.80, 0.72, 0.60]
kept = nms(boxes, scores, iou_threshold=0.5)
print(f"Kept box indices: {kept}")
# Kept box indices: [0, 3]
# Box 0 (car 1, highest confidence) and box 3 (car 2, different location)
Standard NMS hard-removes any box with IoU above the threshold. Soft-NMS instead reduces the score of overlapping boxes by a Gaussian decay — boxes near the winning box get lower scores but aren't eliminated. This handles cases where two objects genuinely overlap (e.g., a person holding an object) better than hard NMS. Most modern detectors use Soft-NMS or learned NMS replacements.
3 Two-Stage Detectors: The R-CNN Family
The R-CNN family dominated object detection from 2014 to 2018. Understanding their evolution is essential for understanding why modern detectors are designed the way they are.
R-CNN (2014): The First Deep Detector
The original R-CNN pipeline: (1) Use selective search to generate ~2,000 region proposals — bounding boxes likely to contain objects. (2) Crop and resize each proposed region to a fixed size (227×227). (3) Run each crop independently through AlexNet to extract features. (4) Classify each feature vector with an SVM. The results were dramatically better than prior methods. The problem: processing 2,000 regions independently through a CNN took 47 seconds per image. Completely unusable for any real application.
Fast R-CNN (2015): Feature Sharing
The key insight: why run the CNN 2,000 times? Run it once on the full image to produce a shared feature map. Then, for each region proposal, use RoI Pooling (Region of Interest Pooling) to extract a fixed-size feature vector from the shared feature map. Classification now operates on these pooled features. The CNN runs once instead of 2,000 times — this reduced inference time from 47 seconds to 2.3 seconds. Still too slow for real-time, but a massive improvement.
Faster R-CNN (2015): Learned Proposals
The remaining bottleneck was selective search — it ran on CPU and couldn't be trained. Faster R-CNN replaced it with a Region Proposal Network (RPN): a small convolutional network that slides over the feature map and, at each position, predicts whether an anchor box contains an object and refines the box coordinates. The RPN shares the backbone with the detector — the whole system is end-to-end trainable. Inference: 0.2 seconds per image (5 FPS). Not real-time, but the foundation of modern two-stage detectors.
import torch
import torchvision
from torchvision.models.detection import fasterrcnn_resnet50_fpn
from torchvision.models.detection import FasterRCNN_ResNet50_FPN_Weights
# Load Faster R-CNN pretrained on COCO (80 classes)
model = fasterrcnn_resnet50_fpn(
weights=FasterRCNN_ResNet50_FPN_Weights.DEFAULT
)
model.eval()
# Run inference on a dummy image (batch of 1)
dummy_image = torch.rand(3, 600, 800) # C x H x W
with torch.no_grad():
predictions = model([dummy_image])
pred = predictions[0]
print(f"Boxes: {pred['boxes'].shape}") # (N, 4) detected boxes
print(f"Labels: {pred['labels'].shape}") # (N,) class indices
print(f"Scores: {pred['scores'].shape}") # (N,) confidence scores
# Filter to high-confidence detections
high_conf = pred['scores'] > 0.5
boxes = pred['boxes'][high_conf]
labels = pred['labels'][high_conf]
scores = pred['scores'][high_conf]
print(f"High-confidence detections: {boxes.shape[0]}")
Anchors are pre-defined bounding boxes of different sizes and aspect ratios that cover the image at each position of the feature map. Instead of predicting absolute box coordinates, the detector predicts offsets from the nearest anchor. Typical anchors: 3 scales × 3 aspect ratios = 9 anchors per position. For a 50×50 feature map, that is 50×50×9 = 22,500 candidate boxes. Anchors make training more stable by giving the network a good starting point for each prediction.
4 One-Stage Detectors: YOLO
Two-stage detectors are accurate but slow. The fundamental reason: stage one proposes regions, stage two classifies them — two forward passes (or one with separate heads) plus proposal generation overhead. One-stage detectors eliminate this by predicting boxes and classes simultaneously in a single forward pass.
YOLO v1 (2016): You Only Look Once
YOLO's key idea: divide the image into an S×S grid (e.g., 7×7). Each grid cell predicts B bounding boxes (e.g., 2) and class probabilities. All predictions happen simultaneously in one forward pass. The result: 45 FPS on a GPU versus 5 FPS for Faster R-CNN. The tradeoff was accuracy — early YOLO struggled with small objects and objects in groups — but for real-time applications, 45 FPS at decent accuracy was revolutionary.
YOLOv5 and YOLOv8: Modern YOLO
The YOLO family has iterated rapidly. YOLOv5 (Ultralytics, 2020) introduced a clean PyTorch codebase and became the industry standard for real-time detection. YOLOv8 (2023) further improved accuracy/speed tradeoffs with an anchor-free design and better training recipes. It supports detection, segmentation, pose estimation, and classification in one package.
# Install: pip install ultralytics
from ultralytics import YOLO
from PIL import Image
import requests, io
# Load YOLOv8 nano — smallest, fastest model
model = YOLO('yolov8n.pt') # downloads ~6MB weights
# Run inference on an image URL
results = model('https://ultralytics.com/images/bus.jpg')
# Inspect results
result = results[0]
print(f"Image size: {result.orig_shape}")
print(f"Detected {len(result.boxes)} objects:\n")
for box in result.boxes:
class_id = int(box.cls)
class_name = model.names[class_id]
confidence = float(box.conf)
coords = box.xyxy[0].tolist() # [x_min, y_min, x_max, y_max]
print(f" {class_name:12s} conf={confidence:.2f} box={[int(c) for c in coords]}")
# Save annotated image
result.save('detection_output.jpg')
Comparing YOLO Variants
from ultralytics import YOLO
import time
# Compare different YOLOv8 sizes (nano to extra-large)
models_to_compare = {
'yolov8n': 'Nano (3M params, fastest)',
'yolov8s': 'Small (11M params)',
'yolov8m': 'Medium (26M params)',
'yolov8l': 'Large (44M params)',
'yolov8x': 'Extra-large (68M params, most accurate)',
}
for model_name, description in models_to_compare.items():
print(f"{model_name}: {description}")
# In practice:
# yolov8n: ~300 FPS on GPU (real-time edge deployment)
# yolov8m: ~120 FPS on GPU (balanced production use)
# yolov8x: ~40 FPS on GPU (maximum accuracy)
The YOLO naming is fragmented. The original authors (Redmon) released v1–v3 before leaving the field. YOLOv4 was from a different team. YOLOv5 is Ultralytics (not officially from the original paper). YOLOv6, YOLOv7, and YOLOv9 are all from different teams. For practical work, stick with YOLOv8 or YOLOv10 from Ultralytics — they have the best documentation, tooling, and community support.
5 Feature Pyramid Networks (FPN)
One of the most persistent challenges in object detection is scale variation. A car far away might occupy a 30×30 pixel region; the same car close up occupies 400×400 pixels. A single feature map extracted from a deep CNN has high-level semantic information (knows what things are) but poor spatial resolution (has been downsampled many times). It struggles with small objects because those objects have been compressed away.
The Problem with Single-Scale Features
Deep networks downsample the feature map at each stage. An input 640×640 image becomes a 20×20 feature map after 5 downsampling stages. A small pedestrian that was 40×40 pixels (6% of image width) is now just 1–2 positions on the feature map. The spatial detail needed to detect it has been lost to pooling.
FPN: A Hierarchy of Feature Maps
Feature Pyramid Networks (Lin et al., 2017) solve this with two pathways: the bottom-up pathway is the normal forward pass of the backbone CNN, producing feature maps at multiple scales (progressively smaller, but richer in semantics). The top-down pathway starts from the smallest, richest feature map and progressively upsamples it back to larger resolutions, merging with the corresponding bottom-up feature maps via lateral (1×1) connections. The result: feature maps at multiple scales that each have both spatial detail (from shallow layers) and semantic richness (from deep layers). Different scales detect objects of different sizes.
import torch
import torch.nn as nn
import torch.nn.functional as F
class SimpleFPN(nn.Module):
"""A minimal Feature Pyramid Network to illustrate the concept."""
def __init__(self, in_channels_list, out_channels=256):
"""
in_channels_list: number of channels at each backbone stage,
e.g., [256, 512, 1024, 2048] for ResNet
out_channels: unified channel count for all pyramid levels
"""
super().__init__()
# Lateral convolutions: 1x1 to unify channel counts
self.lateral_convs = nn.ModuleList([
nn.Conv2d(in_ch, out_channels, kernel_size=1)
for in_ch in in_channels_list
])
# Output convolutions: 3x3 to smooth the merged features
self.output_convs = nn.ModuleList([
nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1)
for _ in in_channels_list
])
def forward(self, features):
"""
features: list of feature maps from backbone stages
[C2, C3, C4, C5] from large to small resolution
Returns: list of FPN feature maps [P2, P3, P4, P5]
"""
# Apply lateral convolutions to all stages
laterals = [conv(f) for conv, f in zip(self.lateral_convs, features)]
# Top-down pathway: merge from deepest to shallowest
for i in range(len(laterals) - 2, -1, -1):
# Upsample deeper layer and add to shallower layer
upsampled = F.interpolate(laterals[i + 1],
size=laterals[i].shape[-2:],
mode='nearest')
laterals[i] = laterals[i] + upsampled
# Apply output convolutions
outputs = [conv(lat) for conv, lat in zip(self.output_convs, laterals)]
return outputs
# Simulate backbone feature maps from ResNet stages
batch_size = 2
backbone_features = [
torch.randn(batch_size, 256, 80, 80), # C2: large, shallow
torch.randn(batch_size, 512, 40, 40), # C3
torch.randn(batch_size, 1024, 20, 20), # C4
torch.randn(batch_size, 2048, 10, 10), # C5: small, deep
]
fpn = SimpleFPN(in_channels_list=[256, 512, 1024, 2048], out_channels=256)
pyramid_features = fpn(backbone_features)
for i, feat in enumerate(pyramid_features):
print(f"P{i+2}: shape {feat.shape} — detects {'large' if i < 2 else 'small'} objects")
FPN is used in virtually every state-of-the-art detector: Faster R-CNN with FPN, RetinaNet, YOLOv5/v8, DETR variants, and more. The consistent output channel dimension (256) across all pyramid levels means the detection head is shared — the same head processes features from all scales, making the model parameter-efficient. When you load fasterrcnn_resnet50_fpn from torchvision, the "fpn" in the name refers to exactly this architecture.
6 Semantic Segmentation: FCN and U-Net
Semantic segmentation requires predicting a class label for every pixel in the image. The output is not a set of bounding boxes but a dense label map of the same spatial dimensions as the input. This demands architectures that can produce spatially precise outputs.
Fully Convolutional Networks (FCN, 2015)
The key insight: replace the fully connected layers at the end of a classification CNN with 1×1 convolutions. A traditional CNN classifier compresses the spatial dimensions down to a global vector (e.g., 7×7×512 → 25,088 → 4,096 → 1,000). This destroys all spatial information. By using only convolutions, the network can operate on images of any size and produce a spatially-varying output map. Transposed convolutions (sometimes called deconvolutions) upsample this map back to the original image resolution. FCN was the first successful end-to-end CNN for dense prediction, achieving near-real-time semantic segmentation.
U-Net (2015): Skip Connections for Spatial Precision
FCN's main weakness: the upsampled output is blurry and imprecise, because fine spatial detail was lost during downsampling. U-Net, developed specifically for biomedical image segmentation, solves this with skip connections: the feature maps from each encoder stage are concatenated with the corresponding decoder stage. This allows the decoder to recover fine-grained spatial information that was lost during pooling. The result is much sharper segmentation boundaries — critical for medical applications where a few pixels can determine whether a tumour margin is correctly identified.
import torch
import torch.nn as nn
import torch.nn.functional as F
class DoubleConv(nn.Module):
"""Two consecutive (Conv → BN → ReLU) blocks — the basic U-Net unit."""
def __init__(self, in_channels, out_channels):
super().__init__()
self.block = nn.Sequential(
nn.Conv2d(in_channels, out_channels, 3, padding=1, bias=False),
nn.BatchNorm2d(out_channels),
nn.ReLU(inplace=True),
nn.Conv2d(out_channels, out_channels, 3, padding=1, bias=False),
nn.BatchNorm2d(out_channels),
nn.ReLU(inplace=True),
)
def forward(self, x):
return self.block(x)
class UNet(nn.Module):
"""
Standard U-Net for semantic segmentation.
Encoder path: downsample with MaxPool
Decoder path: upsample with bilinear interpolation + skip connections
"""
def __init__(self, in_channels=3, num_classes=2, features=[64, 128, 256, 512]):
super().__init__()
# Encoder: series of DoubleConv + MaxPool
self.encoders = nn.ModuleList()
self.pools = nn.ModuleList()
in_ch = in_channels
for feat in features:
self.encoders.append(DoubleConv(in_ch, feat))
self.pools.append(nn.MaxPool2d(2, 2))
in_ch = feat
# Bottleneck
self.bottleneck = DoubleConv(features[-1], features[-1] * 2)
# Decoder: upsample + concatenate skip + DoubleConv
self.upconvs = nn.ModuleList()
self.decoders = nn.ModuleList()
for feat in reversed(features):
# Upsample from features[-1]*2 → feat
self.upconvs.append(
nn.ConvTranspose2d(feat * 2, feat, kernel_size=2, stride=2)
)
# After concatenation with skip: feat * 2 input channels
self.decoders.append(DoubleConv(feat * 2, feat))
# Final 1×1 conv: produce per-pixel class scores
self.output_conv = nn.Conv2d(features[0], num_classes, 1)
def forward(self, x):
skip_connections = []
# Encoder path
for enc, pool in zip(self.encoders, self.pools):
x = enc(x)
skip_connections.append(x)
x = pool(x)
x = self.bottleneck(x)
# Decoder path (reversed)
skip_connections.reverse()
for up, dec, skip in zip(self.upconvs, self.decoders, skip_connections):
x = up(x)
# Handle case where spatial dims don't match exactly
if x.shape != skip.shape:
x = F.interpolate(x, size=skip.shape[2:])
x = torch.cat([skip, x], dim=1) # concatenate along channel dim
x = dec(x)
return self.output_conv(x)
# Test U-Net
model = UNet(in_channels=3, num_classes=5) # 5-class segmentation
total_params = sum(p.numel() for p in model.parameters()) / 1e6
print(f"U-Net parameters: {total_params:.1f}M")
x = torch.randn(2, 3, 256, 256) # batch of 2, RGB, 256×256
logits = model(x)
print(f"Input: {x.shape}")
print(f"Output: {logits.shape}") # (2, 5, 256, 256) — per-pixel class scores
In a standard encoder-decoder (FCN), the decoder only receives the highly compressed bottleneck features. Skip connections let the decoder also see the original high-resolution features at each scale. Think of it like solving a jigsaw puzzle: the deep features tell you what the pieces are (cars, people, road), but the shallow skip features tell you where exactly the boundaries are (these 3 pixels are the edge of the car door). Without skip connections, you know what is in the image but can't draw precise boundaries.
7 Mask R-CNN: Instance Segmentation
Mask R-CNN (He et al., 2017) extends Faster R-CNN with a third output head that produces a binary pixel mask for each detected object instance. It is the canonical instance segmentation architecture and one of the most elegant extensions of a base detection model.
Architecture
Mask R-CNN adds a small FCN (Fully Convolutional Network) head to each RoI (Region of Interest). For each detected object, this FCN predicts a K×K binary mask (one per class), where K is typically 28. The mask head operates in parallel with the classification and box regression heads — it adds only modest computational overhead while enabling full instance segmentation.
RoI Align: Fixing the Misalignment Problem
Faster R-CNN uses RoI Pooling, which involves quantization (rounding pixel coordinates to integers). This causes small misalignments — acceptable for bounding boxes but fatal for pixel-level masks. Mask R-CNN replaces RoI Pooling with RoI Align, which uses bilinear interpolation to extract features at exact (non-integer) locations. This removes quantization artifacts and improves mask quality significantly.
import torch
import torchvision
from torchvision.models.detection import maskrcnn_resnet50_fpn
from torchvision.models.detection import MaskRCNN_ResNet50_FPN_Weights
# Load Mask R-CNN pretrained on COCO
model = maskrcnn_resnet50_fpn(weights=MaskRCNN_ResNet50_FPN_Weights.DEFAULT)
model.eval()
# COCO class names (80 classes)
COCO_CLASSES = [
'__background__', 'person', 'bicycle', 'car', 'motorcycle', 'airplane',
'bus', 'train', 'truck', 'boat', 'traffic light', 'fire hydrant',
'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse',
'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe',
# ... 80 total
]
# Inference
image_tensor = torch.rand(3, 480, 640) # dummy C x H x W image
with torch.no_grad():
predictions = model([image_tensor])
pred = predictions[0]
# pred contains: 'boxes', 'labels', 'scores', 'masks'
high_conf = pred['scores'] > 0.7
masks = pred['masks'][high_conf] # shape: (N, 1, H, W) — float masks
boxes = pred['boxes'][high_conf] # shape: (N, 4)
labels = pred['labels'][high_conf] # shape: (N,)
scores = pred['scores'][high_conf] # shape: (N,)
print(f"Detected {len(boxes)} high-confidence instances")
print(f"Mask tensor shape: {masks.shape}") # (N, 1, 480, 640)
# Convert soft masks to binary
binary_masks = (masks > 0.5).squeeze(1) # (N, H, W) boolean
print(f"Binary mask shape: {binary_masks.shape}")
# Visualize with matplotlib (conceptual)
for i, (box, label_id, score) in enumerate(zip(boxes, labels, scores)):
class_name = COCO_CLASSES[label_id] if label_id < len(COCO_CLASSES) else 'unknown'
print(f" Instance {i}: {class_name} (score={score:.2f})")
Facebook/Meta originally developed Mask R-CNN for automated photo tagging and content understanding. Instagram uses it to detect objects in stories for sticker suggestions and AR filters. In robotics, Mask R-CNN tells a robotic arm exactly which pixels belong to the object it should grasp. In construction site safety, it identifies whether workers are wearing hard hats and safety vests at the pixel level, generating precise compliance reports.
8 Using Pre-trained Detection Models and Evaluating with mAP
Beyond torchvision, the Hugging Face ecosystem provides easy access to state-of-the-art detectors including DETR, which takes a radically different approach.
DETR: Detection Transformer
DETR (Carion et al., 2020) replaces the entire anchor/NMS pipeline with a transformer. The image features are passed to a transformer encoder; a fixed set of learned "object queries" attend to the image features in the decoder; each query predicts exactly one object (or "no object"). No anchors, no NMS, fully end-to-end. The architecture is simpler but training is slower (requires many more epochs to converge).
from transformers import pipeline, DetrImageProcessor, DetrForObjectDetection
from PIL import Image
import requests, torch
# Option 1: High-level pipeline (easiest)
detector = pipeline("object-detection", model="facebook/detr-resnet-50")
# Load an image
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
# Detect objects
results = detector(image)
for obj in results:
label = obj['label']
score = obj['score']
box = obj['box']
print(f" {label:12s} score={score:.2f} box={box}")
# Option 2: Manual DETR (more control)
processor = DetrImageProcessor.from_pretrained("facebook/detr-resnet-50")
model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50")
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
# Convert outputs to bounding boxes
target_sizes = torch.tensor([image.size[::-1]]) # (H, W)
detr_results = processor.post_process_object_detection(
outputs, threshold=0.9, target_sizes=target_sizes
)[0]
print(f"\nDETR high-confidence detections: {len(detr_results['boxes'])}")
Evaluating Detectors: mAP
Mean Average Precision (mAP) is the standard metric for object detection. For each class, compute the precision-recall curve by varying the confidence threshold. Average Precision (AP) is the area under this curve. mAP is the mean AP across all classes. At COCO, the primary metric is mAP averaged over IoU thresholds from 0.5 to 0.95 (reported as mAP@[.5:.95] or simply AP). This is much stricter than mAP@0.5 — it rewards precise localization.
# Model comparison on COCO val2017
# (representative published numbers — actual results vary with training setup)
detection_benchmarks = {
"Faster R-CNN (R50-FPN)": {"mAP@0.5": 58.1, "mAP@0.5:0.95": 37.0, "FPS": 15},
"YOLOv8n": {"mAP@0.5": 52.9, "mAP@0.5:0.95": 37.3, "FPS": 312},
"YOLOv8x": {"mAP@0.5": 68.9, "mAP@0.5:0.95": 53.9, "FPS": 52},
"DETR (R50)": {"mAP@0.5": 63.3, "mAP@0.5:0.95": 42.0, "FPS": 28},
}
print(f"{'Model':<30} {'mAP@0.5':>8} {'mAP@.5:.95':>12} {'FPS':>6}")
print("-" * 60)
for model_name, metrics in detection_benchmarks.items():
print(f"{model_name:<30} {metrics['mAP@0.5']:>8.1f} "
f"{metrics['mAP@0.5:0.95']:>12.1f} {metrics['FPS']:>6}")
mAP@0.5 is permissive — any box with IoU ≥ 0.5 with the ground truth counts as a correct detection. mAP@0.5:0.95 (also written AP or COCO AP) averages over 10 IoU thresholds from 0.50 to 0.95 in 0.05 steps. This rewards not just finding the object but also precisely localizing it. A model that finds every car but draws the box 30% too large will score well on mAP@0.5 but poorly on mAP@0.5:0.95. Always report both when publishing results.
Real-World Spotlight: Autonomous Driving Perception and Medical Imaging
Object detection and segmentation are not academic exercises — they are at the core of two of the most consequential AI applications: autonomous vehicles and medical imaging.
Autonomous Driving: Five Perception Tasks Simultaneously
A self-driving car's perception system runs multiple tasks on each camera frame (30+ FPS):
- Semantic segmentation: classifies every pixel as road / sidewalk / vehicle / pedestrian / building / sky — provides the driveable area map
- Object detection: bounding boxes with 3D distance estimates for cars, pedestrians, cyclists, traffic lights
- Lane detection: detects lane markings as polylines
- Traffic sign recognition: reads sign content (speed limits, stop signs)
- Depth estimation: per-pixel depth from mono or stereo cameras
These tasks share a backbone (usually a ResNet or EfficientNet with FPN) and have separate lightweight heads. The entire inference must complete within 33ms (30 FPS) for real-time control.
Medical Imaging: U-Net for Tumour Segmentation
In radiology, U-Net (and its 3D extension for volumetric MRI data) segments tumours, organs, and lesions at the pixel level. Each pixel is labeled as tumour / healthy tissue. This is substantially different from detection with bounding boxes: the exact shape of a tumour margin matters for radiation therapy planning — too small a margin and cancer cells survive, too large and healthy tissue is irradiated.
# Illustrative training loop for medical image segmentation with U-Net
import torch
import torch.nn as nn
import torch.optim as optim
# Loss: Dice loss is preferred over cross-entropy for class-imbalanced segmentation
# (tumour pixels are rare — usually < 5% of image pixels)
class DiceLoss(nn.Module):
def __init__(self, smooth=1.0):
super().__init__()
self.smooth = smooth
def forward(self, pred, target):
# pred: (B, num_classes, H, W) logits
# target: (B, H, W) integer class labels
pred = torch.softmax(pred, dim=1)
target_one_hot = torch.zeros_like(pred).scatter_(
1, target.unsqueeze(1), 1.0
)
# Compute Dice per class, then average
dims = (0, 2, 3)
intersection = (pred * target_one_hot).sum(dims)
cardinality = (pred + target_one_hot).sum(dims)
dice_per_class = (2 * intersection + self.smooth) / (cardinality + self.smooth)
return 1 - dice_per_class.mean()
# Why Dice loss? Example:
# Image: 256x256 = 65,536 pixels; tumour: 500 pixels (0.76%)
# A model predicting "all background" would have 99.24% pixel accuracy
# but 0% sensitivity to the tumour — useless clinically!
# Dice loss equally weights foreground and background by measuring overlap ratio
# False negative: missed tumour → high clinical cost
# False positive: extra biopsy → lower clinical cost (but still undesirable)
print("Dice loss handles class imbalance better than cross-entropy for segmentation")
print("Typical Dice scores: random init=0.0, baseline CNN=0.72, U-Net=0.86, U-Net+=0.91")
In lung cancer radiotherapy, U-Net segments the gross tumour volume (GTV) in CT scans. A false negative (missing tumour pixels) means underdosing cancer cells that will regrow. A false positive (irradiating healthy tissue) causes collateral damage to the heart, spine, or oesophagus. The clinical standard for inter-radiologist agreement is a Dice score of ~0.80–0.85. AI models trained on thousands of annotated scans now routinely achieve Dice scores of 0.88–0.92 — surpassing average inter-observer agreement. This reduces manual contouring time from 45 minutes per patient to 5 minutes for review-and-edit.
✍️ Practice Exercises
- Implement the
compute_ioufunction and verify: (a) two identical boxes → IoU=1.0, (b) non-overlapping boxes → IoU=0.0, (c) a box that contains another box exactly → verify the formula. Then test your NMS implementation on a set of 8 boxes, 4 of which overlap the same object. - Load Faster R-CNN (
fasterrcnn_resnet50_fpn) from torchvision, run it on a real image from COCO, and print all detected objects with confidence > 0.7 alongside their COCO class names. - Instantiate the U-Net from this lesson with
num_classes=2and input size 512×512. Count trainable parameters. Modify the architecture to add dropout (p=0.2) before each decoder DoubleConv block to reduce overfitting on small medical datasets. - Use the Ultralytics YOLO library to run YOLOv8n and YOLOv8x on the same image. Compare the detected objects, confidence scores, and inference times. What additional objects does the larger model detect?
▶ Show Solution (Exercise 1 — IoU Edge Cases)
def compute_iou(box1, box2):
x1 = max(box1[0], box2[0]); y1 = max(box1[1], box2[1])
x2 = min(box1[2], box2[2]); y2 = min(box1[3], box2[3])
inter = max(0, x2-x1) * max(0, y2-y1)
a1 = (box1[2]-box1[0]) * (box1[3]-box1[1])
a2 = (box2[2]-box2[0]) * (box2[3]-box2[1])
union = a1 + a2 - inter
return inter / union if union > 0 else 0.0
b1 = [0, 0, 100, 100]
b2 = [0, 0, 100, 100]
print(f"Identical: IoU = {compute_iou(b1, b2):.3f}") # 1.000
b3 = [200, 200, 300, 300]
print(f"No overlap: IoU = {compute_iou(b1, b3):.3f}") # 0.000
b4 = [25, 25, 75, 75] # b4 is inside b1
# intersection = 50x50 = 2500; b1=10000, b4=2500; union=12500-2500=10000
print(f"b4 inside b1: IoU = {compute_iou(b1, b4):.3f}") # 0.250
📚 Primary Source for This Lesson
Redmon, Divvala, Girshick & Farhadi (2016) — "You Only Look Once: Unified, Real-Time Object Detection"
The paper that introduced single-pass (one-stage) object detection. For two-stage detectors, see Ren et al. (2015) "Faster R-CNN"; for semantic segmentation, see Long, Shelhamer & Darrell (2015) "Fully Convolutional Networks for Semantic Segmentation."