Transfer Learning: ResNet, EfficientNet & MobileNet — ML & DL Mastery

🎯 What You'll Learn

Understand why transfer learning works and when to use feature extraction vs fine-tuning
Freeze backbone parameters with requires_grad = False and replace the classifier head
Fine-tune the last few layers of a pre-trained ResNet with a small learning rate
Understand ResNet's residual connections and why they solve the degradation problem
Use EfficientNet and MobileNet from torchvision.models for different size/accuracy tradeoffs
Write a reusable function that adapts any torchvision model to a custom number of classes
Apply the full transfer learning workflow on a small medical imaging dataset

💡

The Big Intuition

Imagine learning to drive in one country and then visiting another country where they drive on the opposite side. You don't start from scratch — you transfer almost everything you know (steering, braking, road rules) and only re-learn a small part (which side to drive on). Transfer learning in deep learning is the same idea: a model trained on millions of images has already learned to detect edges, textures, shapes, and objects. You take that knowledge and adapt just the final layers to your specific task.

1 Why Transfer Learning Is Revolutionary

Training a deep CNN from scratch on a large dataset requires enormous resources. Consider ResNet-50 trained on ImageNet:

1.28 million labeled training images across 1,000 classes
~90 GPU-hours on a V100 GPU (≈ $50–100 in cloud compute)
25 million parameters that need careful optimization
Months of data collection and annotation effort

Most practitioners and researchers don't have this. But here's the remarkable thing: you don't need it. The representations learned by ResNet-50 on ImageNet — edges, textures, shapes, object parts — are useful for almost any image recognition task. Cats, medical scans, satellite imagery, product photos — they all share the same low-level structure that the early layers of ImageNet-trained models have already mastered.

The Two Transfer Learning Strategies

Strategy	What's Frozen	What's Trained	Best For
Feature Extraction	Entire backbone	New head only	Small datasets (<500 images), quick experiments
Fine-tuning	Early layers only (optional)	Last few blocks + new head	Medium datasets (1k–100k images), target domain differs from ImageNet

🌍

How Much Does Transfer Learning Actually Help?

A widely cited study (Kornblith et al., 2019) compared training from scratch vs transfer learning across 12 datasets with varying sizes. Transfer learning consistently outperformed training from scratch, especially with small datasets. With 200 labeled images: training from scratch might achieve 50–60%; transfer learning routinely achieves 85–95%. The pre-trained features are so powerful that even a linear classifier on top achieves excellent results — no fine-tuning needed.

The Core Decision: What to Freeze, What to Train

Every transfer learning project starts with the same choice: how much of the pretrained network do you let training touch? Click each strategy below to see which layers of a pretrained backbone stay frozen (locked, weights fixed) and which become trainable (unlocked, weights update via backpropagation).

Feature Extraction: the entire pretrained backbone is frozen — only the new classifier head is trained. Fastest option, best for small datasets.

2 Feature Extraction: Using a CNN as a Fixed Feature Extractor

In feature extraction mode, we freeze all backbone parameters — they receive no gradient updates and don't change during training. We only add and train a new final layer (the "head") that maps from the backbone's output features to our custom classes.

Step 1: Load a Pre-trained Model

In [1]:

import torch
import torch.nn as nn
import torchvision.models as models

# Load ResNet-18 pre-trained on ImageNet
# torchvision 0.13+: use weights= instead of pretrained=True
model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)

# Inspect the final layer
print(model.fc)
# Linear(in_features=512, out_features=1000, bias=True)
# ResNet-18's final fully connected layer: 512 features -> 1000 ImageNet classes

Step 2: Freeze the Backbone

In [2]:

# Freeze ALL parameters in the model
for param in model.parameters():
    param.requires_grad = False

# Verify: no parameters have requires_grad = True
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
frozen = sum(p.numel() for p in model.parameters() if not p.requires_grad)
print(f"Trainable parameters: {trainable:,}")   # 0
print(f"Frozen parameters:    {frozen:,}")       # 11,181,642 (all frozen)

Step 3: Replace the Classifier Head

In [3]:

NUM_CLASSES = 5  # our custom dataset has 5 classes

# Replace the final fully connected layer
# The new layer has requires_grad=True by default (it's newly created)
model.fc = nn.Linear(model.fc.in_features, NUM_CLASSES)
# model.fc.in_features = 512 for ResNet-18

# Now count trainable params again
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Trainable parameters after replacing head: {trainable:,}")
# Only the new head: 512 * 5 + 5 = 2,565 parameters

Step 4: Train Only the Head

In [4]:

import torch.optim as optim

device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = model.to(device)
criterion = nn.CrossEntropyLoss()

# Pass only the trainable parameters to the optimizer
# (passing frozen params is harmless but wasteful — they have no grad)
optimizer = optim.Adam(
    filter(lambda p: p.requires_grad, model.parameters()),
    lr=1e-3
)

# Training is extremely fast — only 2,565 parameters to update
# A frozen backbone means no backprop through 11M parameters
# This reduces training time by ~10x compared to full fine-tuning

💡

When Feature Extraction Is Enough

If your images are similar to ImageNet (natural photographs of objects) and your dataset is small (<1,000 images), feature extraction often works surprisingly well — even without any fine-tuning. The frozen backbone acts as a fixed, very powerful feature extractor. A simple linear layer on top of ResNet-18 features can achieve 85–90% accuracy on many standard benchmarks. If you're doing a quick prototype or don't have a GPU, start here.

3 Fine-Tuning: Adapting Pre-trained Weights

Fine-tuning unfreezes some or all layers and trains them on your dataset. The key principle: use a very small learning rate for the backbone. The pre-trained weights are already excellent — large updates would destroy the valuable representations. The new head, starting from random initialization, needs larger updates.

Why Small Learning Rate for the Backbone?

Think of it like adjusting a precise mechanical clock. The pre-trained weights are the clock's main mechanism, carefully calibrated over weeks. You don't want to apply large forces (high LR) that could strip the gears. You want gentle adjustments (low LR) to fine-tune the mechanism for your specific time zone. The new classifier head is a new component being fitted for the first time — it needs larger, faster adjustments.

Discriminative Learning Rates

Advanced fine-tuning uses different learning rates for different parts of the network — smaller for early layers, larger for later layers. This reflects how much each layer needs to change: early layers learn universal features (edges) that generalize everywhere; later layers learn more task-specific features that might need updating.

In [5]:

import torchvision.models as models
import torch.nn as nn
import torch.optim as optim

model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)
NUM_CLASSES = 10

# Strategy: freeze early layers, unfreeze last residual block + head
# ResNet-18 layers: layer1, layer2, layer3, layer4, fc
# We unfreeze layer3, layer4, and replace fc

# First freeze everything
for param in model.parameters():
    param.requires_grad = False

# Unfreeze layer3, layer4 (last two residual blocks)
for param in model.layer3.parameters():
    param.requires_grad = True
for param in model.layer4.parameters():
    param.requires_grad = True

# Replace and unfreeze the head
model.fc = nn.Linear(512, NUM_CLASSES)
# model.fc is newly created, requires_grad=True by default

device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = model.to(device)

# Discriminative learning rates:
# - Backbone (unfrozen layers): very small LR — gentle updates
# - New head: normal LR — it starts random and needs larger updates
optimizer = optim.AdamW([
    {'params': model.layer3.parameters(), 'lr': 1e-5},
    {'params': model.layer4.parameters(), 'lr': 1e-5},
    {'params': model.fc.parameters(),     'lr': 1e-3},
], weight_decay=1e-4)

trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"Trainable: {trainable:,} / {total:,} ({100*trainable/total:.1f}%)")

Out[5]:

Trainable: 3,680,266 / 11,181,642 (32.9%)

🔑

The Two-Phase Fine-Tuning Recipe

Phase 1 (5–10 epochs): Freeze backbone, train only the new head. This lets the head reach a reasonable state before you start updating the backbone. If you fine-tune backbone with a randomly initialized head, the large gradients from the random head can corrupt the pre-trained backbone weights. Phase 2 (10–20 epochs): Unfreeze the last few blocks, train with very small LR (1e-5) for backbone and slightly larger LR (1e-3) for head. This is the approach used in fast.ai and most production transfer learning pipelines.

4 ResNet: Why Residual Connections Are a Breakthrough

To understand why ResNet matters, you need to understand the problem it solved. Before ResNet (2015), there was a puzzling observation: training deeper networks gave worse results — not just on the test set (which could be overfitting), but on the training set itself. This is the degradation problem.

The Degradation Problem

Why would a 56-layer network do worse than a 20-layer network on the training set? A 56-layer network should be at least as good as a 20-layer network — just add 36 identity layers (layers that pass the input unchanged). If the extra layers did nothing, performance would be identical. But in practice, they hurt. The reason: as depth increases, gradients must propagate through more layers. The repeated matrix multiplications cause them to vanish (shrink to near zero) before they reach the early layers — those early layers stop learning.

The Skip Connection Solution

He et al. (2015) proposed a simple fix: instead of learning a mapping F(x), let the layer learn only the residual H(x) − x = F(x). The output is then F(x) + x, where the "+x" is a direct, shortcut connection that bypasses 2–3 layers.

Standard layer: output = F(x)
Residual layer: output = F(x) + x

Why does this help? Two reasons:

Gradient highway: the identity shortcut provides a direct path for gradients to flow backward. Even if gradients vanish through F(x), they flow unchanged through the "+x" path. Early layers always receive gradients.
Easier to learn identity: it's easier to learn F(x) = 0 (residual = zero, output = input unchanged) than to learn H(x) = x from scratch. If a layer doesn't need to do anything useful, it can simply output zeros and let the shortcut pass x through unchanged.

In [6]:

import torch
import torch.nn as nn

class ResidualBlock(nn.Module):
    """A basic ResNet residual block with two 3x3 convolutions."""
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
        self.bn1   = nn.BatchNorm2d(channels)
        self.relu  = nn.ReLU(inplace=True)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1, bias=False)
        self.bn2   = nn.BatchNorm2d(channels)

    def forward(self, x):
        identity = x          # save input for the skip connection

        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)

        out = out + identity  # residual connection: F(x) + x
        out = self.relu(out)

        return out

# Test the block
block = ResidualBlock(64)
x = torch.randn(2, 64, 16, 16)
out = block(x)
print(f"Input shape:  {x.shape}")    # torch.Size([2, 64, 16, 16])
print(f"Output shape: {out.shape}")  # torch.Size([2, 64, 16, 16])  (same!)

Using Pre-trained ResNets from torchvision

In [7]:

import torchvision.models as models

# Available ResNet variants (pre-trained on ImageNet)
resnet18  = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)   # 11M params
resnet34  = models.resnet34(weights=models.ResNet34_Weights.DEFAULT)   # 21M params
resnet50  = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)   # 25M params
resnet101 = models.resnet101(weights=models.ResNet101_Weights.DEFAULT) # 44M params

# For most tasks: start with ResNet-18 (fast) or ResNet-50 (more accurate)
# print the architecture
print(resnet18)

# Key sizes
for name in ['resnet18', 'resnet50', 'resnet101']:
    m = getattr(models, name)(weights=None)
    n = sum(p.numel() for p in m.parameters())
    print(f"{name:12s}: {n/1e6:.1f}M parameters")

Out[7]:

resnet18 : 11.7M parameters resnet50 : 25.6M parameters resnet101 : 44.5M parameters

5 EfficientNet: Scaling Networks Efficiently

The core insight behind EfficientNet (Tan & Le, 2019) starts with a question: if you want to improve a CNN's accuracy, what do you scale up? You have three choices:

Width: more channels per layer (wider network)
Depth: more layers (deeper network)
Resolution: larger input images (higher resolution)

Prior work scaled these independently. EfficientNet's insight: all three must be scaled together in a balanced way. Doubling depth without increasing resolution and width is wasteful — the deeper layers need higher-resolution inputs to benefit from their capacity. The authors used neural architecture search to find the optimal scaling coefficients and proposed the compound scaling rule:

depth: d = α^φ | width: w = β^φ | resolution: r = γ^φ

EfficientNet-B0 is the baseline; B1–B7 apply increasing values of φ. B7 achieves better accuracy than any previous architecture with significantly fewer parameters.

In [8]:

import torchvision.models as models

# EfficientNet variants — increasingly accurate/expensive
models_info = {
    'efficientnet_b0': models.EfficientNet_B0_Weights.DEFAULT,
    'efficientnet_b2': models.EfficientNet_B2_Weights.DEFAULT,
    'efficientnet_b4': models.EfficientNet_B4_Weights.DEFAULT,
}

for name, weights in models_info.items():
    m = getattr(models, name)(weights=None)
    n = sum(p.numel() for p in m.parameters())
    print(f"{name:20s}: {n/1e6:.1f}M parameters")

# Load and adapt EfficientNet-B0 for our custom task
eff_model = models.efficientnet_b0(
    weights=models.EfficientNet_B0_Weights.DEFAULT
)

# EfficientNet's classifier structure is different from ResNet
print("EfficientNet classifier:")
print(eff_model.classifier)
# Sequential(
#   (0): Dropout(p=0.2, inplace=True)
#   (1): Linear(in_features=1280, out_features=1000, bias=True)
# )

# Replace the classification head (index 1 in the Sequential)
NUM_CLASSES = 5
eff_model.classifier[1] = nn.Linear(1280, NUM_CLASSES)
print(f"\nReplaced head: {eff_model.classifier[1]}")

Out[8]:

efficientnet_b0 : 5.3M parameters efficientnet_b2 : 7.7M parameters efficientnet_b4 : 19.3M parameters EfficientNet classifier: Sequential( (0): Dropout(p=0.2, inplace=True) (1): Linear(in_features=1280, out_features=1000, bias=True) ) Replaced head: Linear(in_features=1280, out_features=5, bias=True)

💡

Depthwise Separable Convolutions: The Efficiency Trick

EfficientNet (and MobileNet) achieve their efficiency through depthwise separable convolutions. A standard 3×3 conv on a 64-channel input costs 3×3×64 = 576 multiplications per output position. A depthwise separable conv splits this into: (1) a 3×3 depthwise conv on each channel separately (9 mults per channel × 64 channels = 576 mults total), plus (2) a 1×1 pointwise conv to mix channels (64 mults per output channel). For 64 output channels, this totals 576 + 64×64 = 4,672 — but the pointwise conv is much faster in practice, giving ~8× speedup vs standard convolution.

6 MobileNet: Efficiency for Edge Devices

MobileNet was designed with a specific constraint: it must run on mobile phones, embedded systems, and IoT devices — hardware with limited compute, memory, and battery. Every architecture decision trades accuracy for efficiency.

MobileNetV2 and V3

MobileNetV2 introduced the inverted residual block: unlike ResNet which compresses then expands (wide → narrow → wide), MobileNetV2 expands then compresses (narrow → wide → narrow). The narrow "bottleneck" connections are the residual connections, keeping the skip connections cheap.

MobileNetV3 added Squeeze-and-Excitation attention: after each conv block, it computes a global average per channel, processes it through a small FC network, and uses the outputs to weight (re-scale) each channel. This lets the network learn "which channels are important for this image" dynamically.

In [9]:

import torchvision.models as models
import torch

# MobileNet variants
mobilenet_v2 = models.mobilenet_v2(weights=models.MobileNet_V2_Weights.DEFAULT)
mobilenet_v3_large = models.mobilenet_v3_large(
    weights=models.MobileNetV3Large_Weights.DEFAULT
)
mobilenet_v3_small = models.mobilenet_v3_small(
    weights=models.MobileNetV3Small_Weights.DEFAULT
)

for name, m in [('MobileNetV2', mobilenet_v2),
                ('MobileNetV3 Large', mobilenet_v3_large),
                ('MobileNetV3 Small', mobilenet_v3_small)]:
    n_params = sum(p.numel() for p in m.parameters()) / 1e6
    # Measure inference time
    x = torch.randn(1, 3, 224, 224)
    print(f"{name:20s}: {n_params:.1f}M params")

# Adapting MobileNetV3 for custom classes
print("\nMobileNetV3 classifier:")
print(mobilenet_v3_large.classifier)
# Sequential(
#   (0): Linear(in_features=960, out_features=1280, bias=True)
#   (1): Hardswish()
#   (2): Dropout(p=0.2)
#   (3): Linear(in_features=1280, out_features=1000, bias=True)
# )

# Replace final linear layer
mobilenet_v3_large.classifier[3] = nn.Linear(1280, NUM_CLASSES)
print(f"\nReplaced: {mobilenet_v3_large.classifier[3]}")

🌍

MobileNet in the Real World

When you use Instagram Stories and it puts a background filter behind you — detecting your body outline in real time — it's running a MobileNet-based segmentation model on your phone's neural engine at 30 frames per second. When Google Photos recognises faces and scenes for search, it uses MobileNet-derived models that run on-device without sending data to servers. When your phone's camera app recognises a QR code or a plant species — MobileNet. The design choices in MobileNet (depthwise separable convs, inverted residuals) are not academic curiosities — they make real-time on-device vision possible.

7 Practical Transfer Learning Workflow

Let's put it all together in a clear, repeatable 7-step workflow. This is the exact process you'd use in a real project.

In [10]:

import torch
import torch.nn as nn
import torch.optim as optim
import torchvision.models as models
import torchvision.transforms as transforms
import torchvision

# ─────────────────────────────────────────────
# STEP 1: Load the pre-trained backbone
# ─────────────────────────────────────────────
model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)

# STEP 2: Inspect the architecture
# print(model)  # see all layers
print(f"Final layer: {model.fc}")
print(f"Input features to final layer: {model.fc.in_features}")  # 512

# ─────────────────────────────────────────────
# STEP 3: Replace the final layer
# ─────────────────────────────────────────────
NUM_CLASSES = 4  # example: 4 custom classes
model.fc = nn.Linear(model.fc.in_features, NUM_CLASSES)

# ─────────────────────────────────────────────
# STEP 4: Freeze the backbone
# ─────────────────────────────────────────────
for name, param in model.named_parameters():
    if 'fc' not in name:        # everything except the new head
        param.requires_grad = False

trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Phase 1 trainable params: {trainable:,}")  # just the head

# ─────────────────────────────────────────────
# STEP 5: Train the head (Phase 1)
# ─────────────────────────────────────────────
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = model.to(device)

optimizer_phase1 = optim.Adam(
    filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3
)
criterion = nn.CrossEntropyLoss()

# (assume trainloader is already prepared)
# run train_epoch() for 5-10 epochs here...
print("Phase 1: Train head for 5-10 epochs at LR=1e-3")

# ─────────────────────────────────────────────
# STEP 6: Unfreeze last layers (Phase 2, optional)
# ─────────────────────────────────────────────
for param in model.layer4.parameters():
    param.requires_grad = True

# ─────────────────────────────────────────────
# STEP 7: Fine-tune with discriminative LRs
# ─────────────────────────────────────────────
optimizer_phase2 = optim.AdamW([
    {'params': model.layer4.parameters(), 'lr': 1e-5},
    {'params': model.fc.parameters(),     'lr': 1e-4},
], weight_decay=1e-4)

scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer_phase2, T_max=15)
print("Phase 2: Fine-tune layer4 + head for 10-15 epochs at LR=1e-5/1e-4")

trainable_p2 = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Phase 2 trainable params: {trainable_p2:,}")

8 Inspecting and Adapting Different Architectures

Every torchvision model has a different internal structure, which means the final layer has a different name and different input features. Knowing how to find and replace it for any architecture is essential.

In [11]:

import torchvision.models as models
import torch.nn as nn

def adapt_model(arch_name, num_classes, pretrained=True):
    """
    Load any torchvision model and replace its final layer
    for a custom classification task.
    Returns the adapted model.
    """
    # Load model (with or without pre-trained weights)
    model_fn = getattr(models, arch_name)
    weights_cls_name = arch_name.replace('_', '').upper() + '_Weights'

    if pretrained:
        try:
            weights_cls = getattr(models, weights_cls_name.title() + 'Weights' if False
                                  else arch_name.replace('resnet', 'ResNet')
                                               .replace('efficientnet_b', 'EfficientNet_B')
                                               .replace('mobilenet_v3_large', 'MobileNetV3Large')
                                               .replace('mobilenet_v2', 'MobileNet_V2')
                                               + '_Weights', None)
        except AttributeError:
            weights_cls = None
        model = model_fn(weights='DEFAULT')
    else:
        model = model_fn(weights=None)

    # Architecture-specific final layer replacement
    if hasattr(model, 'fc'):
        # ResNet, Inception, DenseNet etc.
        in_features = model.fc.in_features
        model.fc = nn.Linear(in_features, num_classes)
        print(f"{arch_name}: replaced model.fc ({in_features} -> {num_classes})")

    elif hasattr(model, 'classifier') and isinstance(model.classifier, nn.Sequential):
        # EfficientNet, MobileNetV3, VGG
        last_layer = model.classifier[-1]
        if isinstance(last_layer, nn.Linear):
            in_features = last_layer.in_features
            model.classifier[-1] = nn.Linear(in_features, num_classes)
            print(f"{arch_name}: replaced model.classifier[-1] ({in_features} -> {num_classes})")

    elif hasattr(model, 'classifier') and isinstance(model.classifier, nn.Linear):
        # Some older models
        in_features = model.classifier.in_features
        model.classifier = nn.Linear(in_features, num_classes)
        print(f"{arch_name}: replaced model.classifier ({in_features} -> {num_classes})")

    return model


# Test with different architectures
for arch in ['resnet18', 'resnet50', 'mobilenet_v2', 'efficientnet_b0']:
    m = adapt_model(arch, num_classes=10)
    n = sum(p.numel() for p in m.parameters()) / 1e6
    print(f"  → {n:.1f}M total params\n")

💡

Quick Architecture Reference

The final layer name for common architectures: ResNet: model.fc (in_features: 512/2048). EfficientNet-B0–B7: model.classifier[1] (in_features: 1280). MobileNetV2: model.classifier[1] (in_features: 1280). MobileNetV3: model.classifier[3] (in_features: 1280). VGG: model.classifier[6] (in_features: 4096). DenseNet: model.classifier (in_features: 1024/1920). When in doubt: print(model) and find the last Linear layer.

🌍

Real-World Spotlight: Medical Image Classification with Minimal Data

Here's a concrete, realistic example of the power of transfer learning. A hospital wants to classify chest X-rays as normal or pneumonia. They have 400 labeled X-rays — far too few to train a CNN from scratch.

In [12]:

import torch, torch.nn as nn, torchvision.models as models
import torchvision.transforms as transforms
import torchvision.datasets as datasets
import torch.optim as optim

# ── Setup ──
device = 'cuda' if torch.cuda.is_available() else 'cpu'
NUM_CLASSES = 2   # normal vs pneumonia

# ── Data: chest X-ray dataset (adapt path to your download) ──
# Dataset: https://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia
# (400 images for this demo — real dataset has ~5000)
train_transform = transforms.Compose([
    transforms.Resize(256),
    transforms.RandomResizedCrop(224, scale=(0.8, 1.0)),
    transforms.RandomHorizontalFlip(),
    transforms.ColorJitter(brightness=0.1, contrast=0.1),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),  # ImageNet stats
])
eval_transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])

# ── Model: Pre-trained ResNet-50 ──
model = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)
model.fc = nn.Linear(model.fc.in_features, NUM_CLASSES)
model = model.to(device)

# ── APPROACH 1: Feature Extraction (freeze backbone) ──
for name, param in model.named_parameters():
    param.requires_grad = 'fc' in name  # only train the head

optimizer = optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3)

# Expected result after 10 epochs:
# - Training from scratch: ~64% accuracy (too little data)
# - Feature extraction (frozen ResNet-50): ~87% accuracy
print("Feature extraction (frozen backbone): ~87% after 10 epochs")

# ── APPROACH 2: Fine-tuning (unfreeze last block) ──
for param in model.parameters():
    param.requires_grad = True  # unfreeze all

optimizer_ft = optim.AdamW([
    {'params': [p for n,p in model.named_parameters() if 'fc' not in n and 'layer4' not in n],
     'lr': 1e-5},
    {'params': model.layer4.parameters(), 'lr': 5e-5},
    {'params': model.fc.parameters(),     'lr': 1e-3},
], weight_decay=1e-4)

# Expected result after 20 epochs (fine-tuned from the feature extraction checkpoint):
# - Fine-tuning last block + head: ~93% accuracy
print("Fine-tuning (last block + head): ~93% after 20 total epochs")
print("Training time on GPU: ~8 minutes (vs ~6 hours from scratch)")
print()
print("Key insight: 400 images + pre-trained ResNet-50  >  100,000 images from scratch")

The accuracy progression from this real-world scenario:

From scratch (400 images): 64% — too few samples, severe overfitting, unreliable
Feature extraction (frozen ResNet-50): 87% — 23 percentage points better, training in 2 minutes
Fine-tuning (last block + head): 93% — another 6 percentage points, still only 8 minutes total

This pattern — transfer learning dramatically outperforming from-scratch training on limited data — is one of the most reproducible results in all of deep learning. It's the reason that in 2024, nearly every production computer vision system uses pre-trained backbones rather than training from scratch.

Validation accuracy on the 400-image chest X-ray task, by strategy.

Total training time to reach that accuracy (log scale — from-scratch takes ~45× longer than feature extraction).

✍️ Practice Exercises

Load ResNet-18 with pre-trained weights. Print the number of trainable and frozen parameters after freezing all layers except layer4 and fc.
Load EfficientNet-B0 and adapt it for 3 classes. Implement the two-phase training loop: 5 epochs with frozen backbone, then 10 epochs with unfrozen last block at LR=1e-5.
Compare inference speed: time a forward pass for (1) ResNet-18, (2) ResNet-50, (3) MobileNetV3-Small on a batch of 32 images of size 224×224. Use torch.cuda.synchronize() for accurate GPU timing.
Implement the adapt_model function from Section 8 and test it on ResNet-50, EfficientNet-B2, and MobileNetV3-Large. Verify the output shape with a dummy input batch.

▶ Show Solution (Exercise 3 — Inference Speed)

In [13]:

import torch, time
import torchvision.models as models

device = 'cuda' if torch.cuda.is_available() else 'cpu'
x = torch.randn(32, 3, 224, 224).to(device)

def measure_inference(model, x, n_warmup=10, n_runs=50):
    model = model.to(device).eval()
    with torch.no_grad():
        for _ in range(n_warmup):   # warmup runs
            _ = model(x)
        if device == 'cuda':
            torch.cuda.synchronize()
        t0 = time.perf_counter()
        for _ in range(n_runs):
            _ = model(x)
        if device == 'cuda':
            torch.cuda.synchronize()
        t1 = time.perf_counter()
    ms_per_batch = (t1 - t0) / n_runs * 1000
    return ms_per_batch

archs = {
    'ResNet-18':         models.resnet18(weights=None),
    'ResNet-50':         models.resnet50(weights=None),
    'MobileNetV3-Small': models.mobilenet_v3_small(weights=None),
}

for name, m in archs.items():
    t = measure_inference(m, x)
    params = sum(p.numel() for p in m.parameters()) / 1e6
    print(f"{name:20s}: {t:.1f} ms/batch ({params:.1f}M params)")

📚 Primary Source for This Lesson

Yosinski, Clune, Bengio & Lipson (2014) — "How transferable are features in deep neural networks?"
The paper that empirically established which CNN layers transfer well and which are task-specific — the basis for this lesson's feature-extraction-vs-fine-tuning guidance. For the ResNet backbone used throughout, see He et al. (2016) "Deep Residual Learning for Image Recognition."

💬 Not sure whether to freeze the backbone or fine-tune it for your dataset size? Describe your dataset (size, similarity to ImageNet) and your AI tutor can recommend a specific transfer-learning strategy.