🎯 What You'll Learn
- Why autoencoders exist: unsupervised representation learning by predicting your own input through an information bottleneck
- The encoder → latent code → decoder architecture, and why the bottleneck's width is the single most important design choice
- How an autoencoder is exactly "nonlinear PCA" — and precisely where a linear autoencoder becomes mathematically identical to PCA
- Denoising and sparse autoencoders — regularizing the bottleneck so it learns robust structure instead of memorizing
- Reconstruction error as an anomaly-detection signal, and how the variational autoencoder (VAE) turns a deterministic bottleneck into the sample-able latent space that GANs and diffusion models build on
1 From PCA to Nonlinear Compression: Why Autoencoders?
Back in Lesson 31, PCA compressed high-dimensional data into a smaller number of components by finding the linear directions of maximum variance. It works — but it has a hard ceiling: PCA can only ever discover straight-line structure. If your data lies on a curved manifold (a spiral, a swiss roll, the space of natural face images), no number of linear components will unfold it without distortion.
An autoencoder asks the same underlying question PCA asks — "what's the smallest number of numbers I need to reconstruct this data?" — but answers it with a neural network instead of an eigendecomposition. Because a neural network with nonlinear activations can represent curved, folded, and arbitrarily complex functions, it can compress data that PCA physically cannot.
import numpy as np
import matplotlib.pyplot as plt
# A nonlinear 1D manifold embedded in 2D -- a curve, not a line.
# This single number "t" fully determines the point: x = t, y = sin(t) + noise.
np.random.seed(42)
t = np.linspace(-3, 3, 300)
X = np.column_stack([t, np.sin(t * 1.5)]) + np.random.normal(0, 0.05, (300, 2))
# PCA's 1 component MUST be a straight line through this 2D space --
# it cannot bend to follow the curve, no matter how the line is rotated.
from sklearn.decomposition import PCA
pca = PCA(n_components=1)
X_pca_1d = pca.fit_transform(X)
X_pca_reconstructed = pca.inverse_transform(X_pca_1d)
pca_error = np.mean(np.sum((X - X_pca_reconstructed) ** 2, axis=1))
print(f"PCA (1 component) reconstruction MSE: {pca_error:.4f}")
# An autoencoder with a 1-unit bottleneck can, in principle, learn to encode
# each point as its true parameter t and decode back through the curve --
# recovering the manifold almost exactly, because nothing forces its
# encoder/decoder functions to be linear.
print("A well-trained 1D-bottleneck autoencoder on this data: MSE approaches 0")
print("-- because the *true* underlying structure genuinely is 1-dimensional,")
print("it's just not a straight line PCA can represent.")If you strip an autoencoder down to a single linear layer for the encoder, a single linear layer for the decoder, no activation functions, and train it with mean-squared-error reconstruction loss, its optimal solution spans exactly the same subspace as PCA's top principal components. Every nonlinear autoencoder you'll build in this lesson is a strict generalization of that linear special case — swap in nonlinear activations, and the bottleneck can represent curved structure PCA cannot touch.
2 The Autoencoder Architecture: Encoder, Bottleneck, Decoder
Every autoencoder has the same three-part shape, and the entire architecture is trained with a single, almost embarrassingly simple objective: reproduce the input.
- Encoder f(x): a network that compresses the input x (dimension d) down to a latent code z (dimension k, where k << d).
- Bottleneck: the latent code z itself — the narrowest layer in the network, and the only channel information is allowed to pass through.
- Decoder g(z): a network that expands the latent code back up to a reconstruction x̂, the same shape as the original input.
Training minimizes reconstruction loss — typically mean squared error for continuous inputs, or binary cross-entropy for pixel values scaled to [0,1] — between x and x̂:
L(x, x̂) = ‖x − g(f(x))‖²
import torch
import torch.nn as nn
class Autoencoder(nn.Module):
def __init__(self, input_dim=784, latent_dim=32):
super().__init__()
# Encoder: input_dim -> ... -> latent_dim (the bottleneck)
self.encoder = nn.Sequential(
nn.Linear(input_dim, 256), nn.ReLU(),
nn.Linear(256, 64), nn.ReLU(),
nn.Linear(64, latent_dim), # no activation -- raw latent code
)
# Decoder: latent_dim -> ... -> input_dim (mirror image of the encoder)
self.decoder = nn.Sequential(
nn.Linear(latent_dim, 64), nn.ReLU(),
nn.Linear(64, 256), nn.ReLU(),
nn.Linear(256, input_dim), nn.Sigmoid(), # pixels scaled to [0, 1]
)
def forward(self, x):
z = self.encoder(x)
x_hat = self.decoder(z)
return x_hat, z
model = Autoencoder(input_dim=784, latent_dim=32) # 784 = 28x28 MNIST pixels
x = torch.rand(16, 784) # a batch of 16 flattened images
x_hat, z = model(x)
print(f"Input shape: {x.shape}")
print(f"Latent shape: {z.shape}") # (16, 32) -- 784 pixels compressed to 32 numbers
print(f"Output shape: {x_hat.shape}") # (16, 784) -- reconstructed back to full size
print(f"Compression ratio: {784 / 32:.1f}x")Notice the training signal is the input itself — x plays the role of both the model's input and its target. This makes autoencoders self-supervised: you get a supervised-style loss function with gradients you can backpropagate, but without a single human-provided label. This is the same trick behind BERT's masked-language-modeling objective (Lesson 54) and GPT's next-token prediction (Lesson 55) — construct a target from the data itself.
See the Bottleneck Effect: Linear (PCA) vs Nonlinear (Autoencoder) Compression
The chart below uses the curved dataset from Section 1. Drag the slider to control how sharply the manifold curves. The blue line is PCA's best possible 1-component reconstruction — always a straight line, because that's all a linear method can produce. The amber curve is what a sufficiently-trained 1-unit-bottleneck autoencoder learns instead — the true nonlinear manifold itself. Watch the reconstruction-error gap widen as curvature increases: that gap is the value nonlinearity adds.
PCA (linear, 1 component) vs an ideal nonlinear 1D-bottleneck autoencoder, reconstructing the same curved data.
3 Training: Reconstruction Loss
Training an autoencoder looks exactly like training any other PyTorch model (Lesson 42) — the only unusual part is that the target labels are the input batch.
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
transform = transforms.Compose([transforms.ToTensor(), transforms.Lambda(lambda x: x.view(-1))])
train_data = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
train_loader = DataLoader(train_data, batch_size=128, shuffle=True)
model = Autoencoder(input_dim=784, latent_dim=32)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.MSELoss()
model.train()
for epoch in range(10):
total_loss = 0.0
for images, _ in train_loader: # note: labels are discarded (_)
images = images.view(images.size(0), -1)
x_hat, z = model(images)
loss = criterion(x_hat, images) # reconstruct the INPUT, not a label
optimizer.zero_grad()
loss.backward()
optimizer.step()
total_loss += loss.item() * images.size(0)
avg_loss = total_loss / len(train_data)
print(f"Epoch {epoch+1}: reconstruction MSE = {avg_loss:.4f}")After training, two things become useful: the decoder can turn any latent vector into a plausible-looking image (even ones you invent), and the encoder gives you a compact, learned feature representation of any input — one you can feed into a downstream classifier, clustering algorithm, or anomaly detector instead of the raw pixels.
4 Undercomplete vs Overcomplete: Why the Bottleneck Matters
The bottleneck width k is the single most consequential hyperparameter in an autoencoder, because it directly controls whether the network is forced to learn anything at all:
- Undercomplete (k < input dimension d): the standard case. The network cannot pass every input through unchanged — it must discard something, which forces it to learn which structure is worth keeping. This is where genuine compression and feature learning happen.
- Overcomplete (k ≥ d): with no bottleneck constraint, the easiest solution is the identity function — literally copy input to output through every layer, achieving zero reconstruction loss while learning nothing useful whatsoever.
import torch
import torch.nn as nn
# An overcomplete autoencoder (latent_dim > input_dim) with NO other constraint
# can trivially learn the identity mapping -- perfect reconstruction, zero
# useful structure learned. This is a genuine failure mode, not a hypothetical.
overcomplete = Autoencoder(input_dim=20, latent_dim=64) # 64 > 20 -- overcomplete!
# If you must use an overcomplete bottleneck (e.g. because you want a very
# expressive latent space for downstream generation), you need an explicit
# regularizer instead of relying on the bottleneck width alone:
# - Sparse autoencoder (Section 6): penalize the number of ACTIVE latent units
# - Denoising autoencoder (Section 5): corrupt the input so copying it verbatim
# is impossible -- the network must learn robust structure to fill the gaps
# - Contractive autoencoder: penalize how sensitive z is to small input changesJust like 100% training accuracy is a red flag for a classifier (Lesson 15), near-zero reconstruction loss on an undercomplete autoencoder that never improves further, or any overcomplete autoencoder at all, can mean the network found a degenerate shortcut rather than learning meaningful structure. Always check what the latent code is actually capturing — visualize it (Section 8, Lesson 32's t-SNE), or verify it's useful for a downstream task — not just how low the loss number is.
5 Denoising Autoencoders
A denoising autoencoder (Vincent et al., 2008) makes one small but powerful change to training: corrupt the input with noise before feeding it to the encoder, but still compute the loss against the clean, uncorrupted original. The network is forced to learn what the data should look like, not how to memorize what it was shown.
import torch
def add_noise(images, noise_factor=0.3):
noisy = images + noise_factor * torch.randn_like(images)
return torch.clamp(noisy, 0.0, 1.0)
model = Autoencoder(input_dim=784, latent_dim=32)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = torch.nn.MSELoss()
model.train()
for epoch in range(10):
for images, _ in train_loader:
images = images.view(images.size(0), -1)
noisy_images = add_noise(images) # corrupt the INPUT only
x_hat, z = model(noisy_images)
loss = criterion(x_hat, images) # compare against the CLEAN target
optimizer.zero_grad()
loss.backward()
optimizer.step()
# At inference time, feed a noisy/corrupted image and get a cleaned-up
# reconstruction -- the network learned the manifold of "plausible" images
# well enough to project a corrupted point back onto it.This single idea — corrupt the input, reconstruct the clean target — is the direct conceptual ancestor of diffusion models' entire training procedure (Lesson 60): both are, at their core, learning to map a noisy sample back toward the clean data manifold. Diffusion models simply repeat this denoising step hundreds of times at carefully controlled noise levels instead of once.
6 Sparse Autoencoders
A sparse autoencoder takes the opposite regularization strategy: allow a wide (even overcomplete) bottleneck, but add a penalty term that pushes most latent units toward zero for any given input, so only a small handful of units are "active" at once. This forces each active unit to specialize — to mean something specific — rather than letting the network spread information diffusely across a wide, unconstrained code.
import torch
import torch.nn as nn
def sparse_loss(z, target_sparsity=0.05, beta=1e-3):
"""KL-divergence penalty pushing average activation toward target_sparsity."""
rho_hat = torch.mean(torch.sigmoid(z), dim=0) # avg activation per unit, across the batch
rho = torch.full_like(rho_hat, target_sparsity)
kl = rho * torch.log(rho / rho_hat) + (1 - rho) * torch.log((1 - rho) / (1 - rho_hat))
return beta * kl.sum()
criterion = nn.MSELoss()
model = Autoencoder(input_dim=784, latent_dim=128) # deliberately wide/overcomplete
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
for images, _ in train_loader:
images = images.view(images.size(0), -1)
x_hat, z = model(images)
loss = criterion(x_hat, images) + sparse_loss(z) # reconstruction + sparsity penalty
optimizer.zero_grad()
loss.backward()
optimizer.step()
break # (one batch shown for illustration)Sparse autoencoders are the technique behind mechanistic-interpretability tools that decompose a large language model's internal activations into human-interpretable "features" — training a sparse autoencoder on a transformer's hidden states, then inspecting which inputs light up each sparse unit.
7 From Autoencoders to Variational Autoencoders (VAEs)
Everything so far produces a deterministic latent code: the same input always maps to the exact same point z, and most of latent space is never visited during training — so decoding a random point you didn't get from encoding a real input usually produces garbage. That's a real problem if your goal is generation rather than compression.
A Variational Autoencoder (VAE) (Kingma & Welling, 2013) fixes this with one architectural change: instead of encoding x to a single point z, the encoder outputs the parameters of a distribution — a mean μ and standard deviation σ — and z is sampled from N(μ, σ²). A KL-divergence loss term additionally pulls every input's distribution toward a standard normal N(0, 1), packing the whole latent space densely and smoothly instead of leaving gaps.
import torch
import torch.nn as nn
class VAE(nn.Module):
def __init__(self, input_dim=784, latent_dim=32):
super().__init__()
self.encoder_body = nn.Sequential(nn.Linear(input_dim, 256), nn.ReLU())
self.fc_mu = nn.Linear(256, latent_dim) # outputs the mean
self.fc_logvar = nn.Linear(256, latent_dim) # outputs log-variance (numerically stable)
self.decoder = nn.Sequential(
nn.Linear(latent_dim, 256), nn.ReLU(),
nn.Linear(256, input_dim), nn.Sigmoid(),
)
def reparameterize(self, mu, logvar):
# The "reparameterization trick": sampling directly is not differentiable,
# so express the random sample as a deterministic function of mu/logvar
# plus external noise -- gradients can now flow through mu and logvar.
std = torch.exp(0.5 * logvar)
eps = torch.randn_like(std)
return mu + eps * std
def forward(self, x):
h = self.encoder_body(x)
mu, logvar = self.fc_mu(h), self.fc_logvar(h)
z = self.reparameterize(mu, logvar)
return self.decoder(z), mu, logvar
def vae_loss(x_hat, x, mu, logvar):
recon_loss = nn.functional.binary_cross_entropy(x_hat, x, reduction='sum')
kl_loss = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp()) # pulls toward N(0,1)
return recon_loss + kl_loss
# Generation: sample directly from the prior N(0, I) and decode -- no input needed
model = VAE()
z_random = torch.randn(8, 32) # 8 random points from the latent prior
generated = model.decoder(z_random)
print(f"Generated {generated.shape[0]} new samples from pure noise, shape {generated.shape}")A standard autoencoder learns to compress; a VAE learns to generate, because its regularized latent space is densely and smoothly populated, so any random draw from N(0, I) decodes to something plausible. That's exactly the property Lesson 59's GAN generator and Lesson 60's diffusion model both need — a latent space you can sample from. The three architectures differ mainly in how they enforce a well-behaved latent space: a VAE does it with an explicit KL penalty during training; a GAN does it implicitly through adversarial competition; a diffusion model does it by learning to reverse a noising process step by step.
8 Practical Applications: Anomaly Detection & Pretraining
Outside of pure generation, an autoencoder's reconstruction error is itself a directly useful signal — a model trained only on normal data will reconstruct normal inputs well and anomalous inputs poorly, because it never learned the structure needed to compress them.
import torch
import numpy as np
# Train the autoencoder ONLY on normal transactions (Lesson 33's anomaly
# detection framing applies directly -- this is an unsupervised detector).
model = Autoencoder(input_dim=20, latent_dim=4)
# ... train on X_normal only, as in Section 3 ...
model.eval()
with torch.no_grad():
x_hat, _ = model(X_test)
reconstruction_error = torch.mean((X_test - x_hat) ** 2, dim=1)
# Anomalies -- fraud, defects, sensor faults -- were never seen during
# training, so the decoder has no learned structure to reconstruct them
# well with; their error will sit well above the normal-data distribution.
threshold = reconstruction_error[:8000].quantile(0.99) # 99th pct of known-normal errors
flagged = reconstruction_error > threshold
print(f"Flagged {flagged.sum().item()} of {len(X_test)} transactions as anomalous")The encoder half also has a life of its own, independent of the decoder: pretrain an autoencoder on a large pool of unlabeled data, discard the decoder, and use the trained encoder as a fixed or fine-tunable feature extractor for a downstream supervised task with far fewer labels than training from scratch would need — the same transfer-learning logic from Lesson 47, applied to representations learned without any labels at all.
| Variant | What it adds | Primary use |
|---|---|---|
| Vanilla (undercomplete) | Bottleneck narrower than input | Nonlinear dimensionality reduction, pretraining |
| Denoising | Corrupted input, clean target | Robust features; image/audio denoising |
| Sparse | Activation-sparsity penalty | Interpretable, disentangled features |
| Variational (VAE) | Probabilistic latent code + KL penalty | Generation — sampling new, plausible data |
Real-World Spotlight: Manufacturing Defect Detection with Autoencoders
A chip fabrication plant produces high-resolution images of every wafer, but defective wafers are extremely rare (well under 1% of production) and defect types keep changing as manufacturing processes evolve — a classic case where labeled examples of "bad" are scarce, but examples of "good" are abundant. This is exactly the setup an autoencoder-based anomaly detector is built for.
import torch
import torch.nn as nn
import numpy as np
class ConvAutoencoder(nn.Module):
"""A convolutional autoencoder for image-shaped defect data."""
def __init__(self):
super().__init__()
self.encoder = nn.Sequential(
nn.Conv2d(1, 16, 3, stride=2, padding=1), nn.ReLU(), # 64x64 -> 32x32
nn.Conv2d(16, 32, 3, stride=2, padding=1), nn.ReLU(), # 32x32 -> 16x16
nn.Conv2d(32, 8, 3, stride=2, padding=1), nn.ReLU(), # 16x16 -> 8x8 (bottleneck)
)
self.decoder = nn.Sequential(
nn.ConvTranspose2d(8, 32, 3, stride=2, padding=1, output_padding=1), nn.ReLU(),
nn.ConvTranspose2d(32, 16, 3, stride=2, padding=1, output_padding=1), nn.ReLU(),
nn.ConvTranspose2d(16, 1, 3, stride=2, padding=1, output_padding=1), nn.Sigmoid(),
)
def forward(self, x):
z = self.encoder(x)
return self.decoder(z), z
# ── Step 1: Train ONLY on images of known-good wafers ──
model = ConvAutoencoder()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.MSELoss(reduction='none') # per-pixel error, not averaged yet
# for epoch in range(30):
# for good_wafer_images in good_wafer_loader: # ONLY non-defective wafers
# x_hat, z = model(good_wafer_images)
# loss = criterion(x_hat, good_wafer_images).mean()
# optimizer.zero_grad(); loss.backward(); optimizer.step()
# ── Step 2: At inference, per-pixel error localizes the defect ──
def detect_and_localize(model, wafer_image, threshold_percentile=99.5, calibration_errors=None):
model.eval()
with torch.no_grad():
x_hat, _ = model(wafer_image)
pixel_error = ((wafer_image - x_hat) ** 2).squeeze() # per-pixel reconstruction error
# A defect-free wafer reconstructs uniformly well everywhere; a scratch,
# particle, or crack is structure the model never learned to compress,
# so it shows up as a bright "error heatmap" at exactly that location.
threshold = np.percentile(calibration_errors, threshold_percentile)
defect_mask = pixel_error > threshold
is_defective = defect_mask.float().mean() > 0.001 # >0.1% of pixels flagged
return is_defective, pixel_error, defect_mask
print("Per-pixel reconstruction error doubles as both a defect flag AND a")
print("localization map -- pointing an inspector directly at the scratch,")
print("contamination, or crack, with zero labeled defect images required.")This is the pattern behind most industrial visual-inspection systems built before large labeled defect datasets exist: train unsupervised on abundant "normal" images, and let reconstruction error double as both a detector and a localizer. As real defect examples accumulate over time, teams typically graduate to a supervised or semi-supervised detector — but the autoencoder baseline ships on day one, with zero defective examples required to get started.
✍️ Practice Exercises
- Train an undercomplete autoencoder (latent_dim=2) on MNIST. Plot the 2D latent codes colored by digit label (no labels used in training — only for the plot afterward). Do digits cluster, even though the model never saw a single label?
- Repeat Exercise 1 with a linear-only autoencoder (no activation functions anywhere) and compare its 2D latent plot to PCA's first two components on the same data. How similar are they, and why does the theory in Section 1 predict this?
- Train a denoising autoencoder on MNIST with
noise_factor=0.5. Feed it test images corrupted with noise it never saw during training (e.g. salt-and-pepper instead of Gaussian). Does it still clean them up reasonably well? What does that suggest about what it actually learned? - Train a vanilla autoencoder only on the digit "1", then compute reconstruction error on a full test set containing all ten digits. Plot the error distribution per digit class — does "1" show the lowest error, confirming the anomaly-detection principle from Section 8?
▶ Hints
import torch
import matplotlib.pyplot as plt
model.eval()
with torch.no_grad():
all_z, all_labels = [], []
for images, labels in test_loader:
images = images.view(images.size(0), -1)
_, z = model(images)
all_z.append(z)
all_labels.append(labels)
all_z = torch.cat(all_z).numpy()
all_labels = torch.cat(all_labels).numpy()
plt.figure(figsize=(8, 6))
scatter = plt.scatter(all_z[:, 0], all_z[:, 1], c=all_labels, cmap='tab10', s=5, alpha=0.6)
plt.colorbar(scatter, label='Digit')
plt.xlabel('Latent dim 1'); plt.ylabel('Latent dim 2')
plt.title('2D Autoencoder Latent Space (unsupervised)')
plt.savefig('ae_latent_space.png', dpi=150)📚 Primary Source for This Lesson
Goodfellow, Bengio & Courville — Deep Learning, Chapter 14: Autoencoders
The canonical reference, covering undercomplete, regularized, and stochastic autoencoders in depth. For denoising autoencoders, see Vincent et al. (2008) "Extracting and Composing Robust Features with Denoising Autoencoders." For the variational autoencoder, see Kingma & Welling (2013) "Auto-Encoding Variational Bayes" — the paper that introduced the reparameterization trick used in Section 7.