🎯 What You'll Learn
- Understand the forward diffusion process: progressively adding noise to images over T timesteps
- Understand the reverse process: training a neural network to predict and remove noise
- See how the U-Net denoiser uses timestep embeddings and attention for image generation
- Understand DDPM vs DDIM sampling and Classifier-Free Guidance (CFG)
- Understand how CLIP text conditioning enables text-to-image generation
- Understand Latent Diffusion Models — the architecture behind Stable Diffusion
- Use the Hugging Face
diffuserslibrary for image generation, img2img, and ControlNet
Imagine dropping a drop of ink into a glass of water. It spreads out, diffuses, until the water is uniformly colored. That's the forward process. Now imagine playing the video backwards: the color magically withdraws, concentrates, becomes the original ink drop. Diffusion models learn to reverse the diffusion process — starting from pure random noise and progressively de-noising it until a clear image appears. The key insight: learning to de-noise is learnable, and learning to de-noise IS learning to generate.
1 From GANs to Diffusion: Why the Shift?
GANs dominated image generation from 2014 to 2021. They produce sharp, realistic images quickly. But they have well-known limitations that became increasingly problematic as quality demands increased.
GAN Limitations at Scale
Training instability (mode collapse, vanishing gradients) meant that getting a GAN to train reliably required careful hyperparameter tuning, architectural tricks, and expert knowledge. Scaling to higher resolutions required increasingly complex solutions (ProGAN's progressive training). Mode coverage was incomplete — GANs tend to produce a subset of the real distribution rather than covering all of it. For content generation at scale (billions of diverse images), this was limiting.
Diffusion Models Emerge (2020–2022)
DDPM (Ho et al., 2020) showed that a simple denoising objective could produce image quality competitive with GANs. More importantly, diffusion models offer several architectural advantages: training is stable (just predicting noise with MSE loss), mode coverage is better (the probabilistic framework naturally covers the full distribution), and conditioning (text, class labels, edge maps) is straightforward to add via cross-attention.
| Property | GAN | Diffusion |
|---|---|---|
| Training stability | Difficult (adversarial) | Stable (simple MSE) |
| Mode coverage | Often incomplete | Good coverage |
| Sampling speed | Fast (1 forward pass) | Slower (many steps) |
| Text conditioning | Hard to add cleanly | Straightforward (cross-attention) |
| Image quality | Sharp but limited diversity | Best overall quality |
2 The Forward Process: Adding Noise Gradually
Diffusion models define a forward process that takes a real image and progressively destroys it by adding Gaussian noise over T timesteps (typically T=1000). By the final timestep, the image has been completely corrupted into pure Gaussian noise — all structure is gone.
The Noise Schedule
At each timestep t, a small amount of noise is added according to a schedule β₁, β₂, ..., β_T, where β_t are small positive values (typically 0.0001 to 0.02):
This says: x_t is a Gaussian with mean √(1−β_t)·x_{t-1} (the previous image, slightly attenuated) and variance β_t. The √(1−β_t) scaling ensures the image doesn't blow up in magnitude — as noise increases, signal decreases proportionally.
The Key Insight: One-Step Noising
Computing x_t requires running t sequential steps of noising. This would make training very slow — you'd need to noise each training image 1000 times. Fortunately, there is a mathematical shortcut: you can jump directly from x₀ to any x_t in a single operation by combining all the noise steps analytically. Let ᾱ_t = ∏_{i=1}^{t}(1 − β_i) (the cumulative product of noise levels). Then:
Equivalently: x_t = √ᾱ_t·x_0 + √(1−ᾱ_t)·ε, where ε ~ N(0, I)
import torch
import numpy as np
class NoiseSchedule:
"""
Linear noise schedule for DDPM.
Precomputes all necessary values for training.
"""
def __init__(self, T=1000, beta_start=1e-4, beta_end=0.02):
self.T = T
# Linear schedule: beta increases from beta_start to beta_end
self.betas = torch.linspace(beta_start, beta_end, T)
self.alphas = 1.0 - self.betas
# Cumulative product: alpha_bar_t = product of alpha_1 through alpha_t
self.alpha_bars = torch.cumprod(self.alphas, dim=0)
# Precompute square roots for efficient sampling
self.sqrt_alpha_bars = torch.sqrt(self.alpha_bars)
self.sqrt_one_minus_abars = torch.sqrt(1.0 - self.alpha_bars)
def add_noise(self, x0, t, noise=None):
"""
Forward process: add noise to x0 at timestep t.
Uses the closed-form formula: x_t = sqrt(alpha_bar_t)*x0 + sqrt(1-alpha_bar_t)*eps
"""
if noise is None:
noise = torch.randn_like(x0)
sqrt_ab = self.sqrt_alpha_bars[t].view(-1, 1, 1, 1)
sqrt_1ab = self.sqrt_one_minus_abars[t].view(-1, 1, 1, 1)
return sqrt_ab * x0 + sqrt_1ab * noise, noise
# Visualize the noise schedule
schedule = NoiseSchedule(T=1000)
print("Noise schedule progression:")
print(f"{'Timestep':>10} {'ᾱ_t (signal strength)':>25} {'√(1-ᾱ_t) (noise level)':>25}")
print("-" * 65)
for t in [0, 100, 250, 500, 750, 999]:
ab = schedule.alpha_bars[t].item()
nlt = schedule.sqrt_one_minus_abars[t].item()
print(f"t={t:>4} {ab:>25.4f} {nlt:>25.4f}")
# Demonstration: noise a dummy image at different timesteps
x0 = torch.randn(1, 3, 64, 64) # dummy "clean" image
for t_val in [100, 500, 999]:
t_tensor = torch.tensor([t_val])
noisy, noise = schedule.add_noise(x0, t_tensor)
signal_ratio = (schedule.alpha_bars[t_val] * x0.var()).item()
print(f"t={t_val}: noisy image variance={noisy.var().item():.3f}")Without the closed-form formula, adding noise from x₀ to x_t would require running t sequential noising steps — for T=1000, that means 1000 steps per training sample per batch. Training on millions of images would be impossibly slow. The closed form x_t = √ᾱ_t·x₀ + √(1−ᾱ_t)·ε lets us jump to any timestep in constant time. We can sample random timesteps during training, making the training process both efficient and randomized (which acts as a form of data augmentation).
Try It: Forward-Noising a Signal with the Real DDPM Formula
The demo below applies the exact closed-form equation from above — x_t = √ᾱ_t·x₀ + √(1−ᾱ_t)·ε — to a tiny 12×12 "image" (a simple heart-shaped pattern of 0s and 1s), using a real noise schedule computed in JavaScript with the same β_start=1e-4, β_end=0.02, T=1000 defaults as the NoiseSchedule class above. Drag the timestep slider to see ᾱ_t drop and the grid dissolve into Gaussian noise exactly as the math predicts — nothing here is a fake "blur" effect, every pixel value is computed from the real formula.
x₀ — the clean signal (t = 0, no noise added yet).
ᾱ_t (signal strength, blue) and √(1−ᾱ_t) (noise level, amber) across all T=1000 timesteps. The marker shows the current t.
Formula in effect right now: x_t = √ᾱ_t · x₀ + √(1−ᾱ_t) · ε, with ε freshly sampled from N(0, I) on every slider move.
3 The Reverse Process: Learning to Denoise
The reverse process is what we actually train. We train a neural network ε_θ(x_t, t) to predict the noise ε that was added to create x_t from x₀. Once the network can predict the noise, it can remove it — iteratively de-noising from x_T to x₀.
The Training Objective
The DDPM training loss is beautifully simple:
In plain English: sample a real image x₀, sample a random timestep t, sample noise ε, compute noisy x_t, pass through the network, predict ε, compute MSE between predicted and actual noise. That's it. No adversarial training, no complex losses — just MSE on noise prediction.
The Reverse Sampling Loop
At inference time, we start from pure noise x_T ~ N(0, I) and iteratively denoise:
import torch
import torch.nn as nn
def ddpm_training_step(model, noise_schedule, x0, device):
"""
One DDPM training step.
Returns the MSE loss between predicted and actual noise.
"""
batch_size = x0.size(0)
# Sample random timesteps for this batch
t = torch.randint(0, noise_schedule.T, (batch_size,), device=device)
# Sample noise
noise = torch.randn_like(x0)
# Forward process: add noise to x0 at timestep t
x_t, _ = noise_schedule.add_noise(x0.to(device), t, noise.to(device))
# Predict the noise
noise_pred = model(x_t, t) # (B, C, H, W)
# MSE loss: predict the exact noise that was added
loss = nn.functional.mse_loss(noise_pred, noise.to(device))
return loss
@torch.no_grad()
def ddpm_sample(model, noise_schedule, shape, device):
"""
DDPM sampling: start from noise, iteratively denoise.
shape: (batch_size, channels, height, width)
Returns the generated image x0.
"""
model.eval()
schedule = noise_schedule
# Start from pure noise
x = torch.randn(shape, device=device)
# Reverse diffusion: T steps from pure noise to clean image
for t in reversed(range(schedule.T)):
t_batch = torch.full((shape[0],), t, device=device, dtype=torch.long)
# Predict noise at this timestep
eps_pred = model(x, t_batch)
# Compute the denoised estimate
alpha_t = schedule.alphas[t].to(device)
alpha_bar = schedule.alpha_bars[t].to(device)
beta_t = schedule.betas[t].to(device)
# Mean of reverse process
coeff = (1 - alpha_t) / torch.sqrt(1 - alpha_bar)
x_prev_mean = (1 / torch.sqrt(alpha_t)) * (x - coeff * eps_pred)
if t > 0:
# Add noise for all timesteps except the last
noise = torch.randn_like(x)
sigma_t = torch.sqrt(beta_t)
x = x_prev_mean + sigma_t * noise
else:
x = x_prev_mean # final step: no noise
return x.clamp(-1, 1)
print("DDPM training: randomly sample timestep t, add noise, predict noise, compute MSE")
print("DDPM sampling: 1000 sequential denoising steps, O(T) cost per image")DDPM trains the network to predict the added noise ε. But mathematically equivalent: the network could instead predict the clean image x₀ directly (given x_t and t, predict x₀ = (x_t − √(1−ᾱ_t)·ε) / √ᾱ_t). Both are equivalent — knowing ε is the same as knowing x₀. In practice, noise prediction tends to be better conditioned numerically and produces better results, which is why it became standard. Some newer methods predict a "velocity" v = √ᾱ_t·ε − √(1−ᾱ_t)·x₀, which performs well at all noise levels.
Forward process: a clean image dissolving into pure noise. Reverse process: the learned denoiser pulling it back out, step by step.
4 The U-Net Denoiser
The neural network that powers DDPM is a U-Net — the same architecture we studied for segmentation in Lesson 57. However, the diffusion U-Net is significantly more capable than the basic segmentation U-Net, incorporating attention mechanisms and timestep conditioning.
Why U-Net?
The denoiser takes a noisy image and outputs a noise map of the same shape. This is exactly the type of task U-Net excels at: processing a spatial input (noisy image) and producing a spatial output (noise at each pixel). The skip connections preserve fine spatial detail — important because denoising at small timesteps (little noise) requires precision about individual pixel values.
Timestep Embedding
The network must know what timestep t it is operating at, because the amount and character of noise is different at each timestep. This is done with a sinusoidal position embedding (the same idea as Transformer position encodings), followed by MLP layers to produce a vector that is added to the feature maps at each resolution level.
import torch
import torch.nn as nn
import math
class SinusoidalTimestepEmbedding(nn.Module):
"""
Sinusoidal timestep embedding — same idea as Transformer position encodings.
Maps timestep t (scalar) to a high-dimensional vector.
"""
def __init__(self, dim):
super().__init__()
self.dim = dim
# MLP to expand the embedding
self.mlp = nn.Sequential(
nn.Linear(dim, dim * 4),
nn.SiLU(),
nn.Linear(dim * 4, dim * 4),
)
def forward(self, t):
# t: (B,) integer timesteps
half = self.dim // 2
# Frequencies: 1/(10000^(2i/d)) for i=0,...,d/2-1
freqs = torch.exp(
-math.log(10000) * torch.arange(half, device=t.device) / (half - 1)
)
args = t.float()[:, None] * freqs[None, :] # (B, half)
embedding = torch.cat([torch.sin(args), torch.cos(args)], dim=1) # (B, dim)
return self.mlp(embedding) # (B, dim*4)
class ResBlock(nn.Module):
"""
Residual block with timestep embedding injection.
Used as the basic unit of the diffusion U-Net.
"""
def __init__(self, in_channels, out_channels, time_emb_dim):
super().__init__()
self.conv1 = nn.Conv2d(in_channels, out_channels, 3, padding=1)
self.conv2 = nn.Conv2d(out_channels, out_channels, 3, padding=1)
self.norm1 = nn.GroupNorm(8, in_channels)
self.norm2 = nn.GroupNorm(8, out_channels)
self.act = nn.SiLU()
# Project time embedding to channel dimension
self.time_proj = nn.Linear(time_emb_dim, out_channels)
# Residual connection (1x1 conv if channels change)
self.skip = (nn.Conv2d(in_channels, out_channels, 1)
if in_channels != out_channels else nn.Identity())
def forward(self, x, time_emb):
h = self.act(self.norm1(x))
h = self.conv1(h)
# Inject timestep information: add time embedding to feature map
h = h + self.time_proj(self.act(time_emb))[:, :, None, None]
h = self.act(self.norm2(h))
h = self.conv2(h)
return h + self.skip(x)
# Demonstrate the components
t_emb_module = SinusoidalTimestepEmbedding(dim=128)
t = torch.tensor([100, 500, 999]) # batch of 3 timesteps
emb = t_emb_module(t)
print(f"Timestep embedding shape: {emb.shape}") # (3, 512)
res = ResBlock(64, 128, time_emb_dim=512)
x = torch.randn(3, 64, 32, 32)
out = res(x, emb)
print(f"ResBlock output shape: {out.shape}") # (3, 128, 32, 32)The diffusion U-Net adds multi-head self-attention blocks at the bottleneck and at medium-resolution levels. Self-attention allows each spatial position to attend to every other position — essential for generating globally coherent images. Without attention, a CNN's receptive field is limited: it can only "see" nearby pixels. Attention lets the network ensure that the sky color in the top-left matches the sky color in the top-right, and that a face has symmetric features. This is why diffusion models generate more coherent high-resolution images than pure convolutional architectures.
5 DDPM vs DDIM: Sampling Speed
DDPM's main weakness is sampling speed: generating one image requires 1000 sequential forward passes through the U-Net. On a modern GPU, this takes 30–60 seconds for a high-resolution image — too slow for interactive use.
DDIM: Deterministic Sampling
DDIM (Song et al., 2020) reinterprets the diffusion process as a non-Markovian one that can be sampled with far fewer steps. The key insight: the DDPM training objective doesn't require Markovian transitions. DDIM defines a deterministic sampling process (no added noise in each step) that can produce good images with 20–50 steps instead of 1000. With 50 steps instead of 1000, sampling is 20× faster. With 20 steps, 50× faster with acceptable quality.
Classifier-Free Guidance (CFG)
Text-to-image models use Classifier-Free Guidance to strengthen adherence to the text prompt. The denoiser is run twice per step: once with the text condition and once without (unconditional). The final noise prediction is extrapolated beyond the conditional prediction, away from the unconditional prediction:
A guidance_scale of 1.0 means no guidance (same as conditional). A guidance_scale of 7.5 (Stable Diffusion default) strongly steers generation toward the prompt. Very high values (>15) produce over-saturated, artifact-ridden images. The tradeoff: higher guidance → more faithful to prompt but less diversity and natural variation.
@torch.no_grad()
def ddim_sample(model, noise_schedule, prompt_embedding, guidance_scale=7.5,
n_steps=50, shape=(1, 4, 64, 64), device='cpu'):
"""
DDIM sampling with Classifier-Free Guidance (CFG).
Uses n_steps << T for fast generation.
prompt_embedding: CLIP text encoding of the prompt
"""
model.eval()
T = noise_schedule.T
# Select evenly spaced timesteps (e.g., 999, 979, 959, ..., 19)
step_indices = torch.linspace(0, T - 1, n_steps).long()
timesteps = step_indices.flip(0) # reverse order for denoising
# Start from pure noise in latent space
x = torch.randn(shape, device=device)
for i, t_val in enumerate(timesteps):
t_batch = torch.full((shape[0],), t_val, device=device, dtype=torch.long)
# CFG: run denoiser twice
# 1. With text conditioning
eps_cond = model(x, t_batch, encoder_hidden_states=prompt_embedding)
# 2. With empty/null conditioning (unconditional)
null_embed = torch.zeros_like(prompt_embedding)
eps_uncond = model(x, t_batch, encoder_hidden_states=null_embed)
# Classifier-Free Guidance interpolation
eps_guided = eps_uncond + guidance_scale * (eps_cond - eps_uncond)
# DDIM update step (deterministic)
ab_t = noise_schedule.alpha_bars[t_val]
x0_pred = (x - torch.sqrt(1 - ab_t) * eps_guided) / torch.sqrt(ab_t)
x0_pred = x0_pred.clamp(-1, 1)
# Next noisy state (if not last step)
if i < n_steps - 1:
ab_next = noise_schedule.alpha_bars[timesteps[i + 1]]
x = (torch.sqrt(ab_next) * x0_pred +
torch.sqrt(1 - ab_next) * eps_guided)
else:
x = x0_pred
return x
# Guidance scale effect (conceptual):
guidance_scale_effects = {
1.0: "Barely follows prompt, high diversity",
3.0: "Mild guidance, good balance",
7.5: "Strong guidance (SD default), clear prompt adherence",
15.0: "Very strong, can over-saturate",
20.0: "Extreme, usually produces artifacts",
}
for gs, effect in guidance_scale_effects.items():
print(f" guidance_scale={gs:.1f}: {effect}")Consistency Models (Song et al., 2023) take this further: they train a network that maps any point on a diffusion trajectory directly to x₀ in a single step. In one forward pass, they can generate good images. With 2–3 steps, they match DDIM quality at 50 steps. Consistency distillation trains a student model to mimic the multi-step DDPM path, compressing it into one or few steps. This closes the speed gap between diffusion models and GANs.
6 Text-to-Image: Adding Language Conditioning
How does "a painting of a cat in the style of Van Gogh" become a pixel arrangement? The key bridge is CLIP — a model that jointly understands text and images.
CLIP: Joint Text-Image Embedding
CLIP (Contrastive Language-Image Pre-training, OpenAI 2021) was trained on 400 million (image, caption) pairs scraped from the internet. Its objective: an image encoder and a text encoder should produce similar embedding vectors for matching pairs and dissimilar vectors for non-matching pairs (contrastive learning). After training, CLIP has a joint embedding space: the vector for "a photo of a dog" is geometrically close to the vector for actual dog photos, and far from cat photos or car photos.
How Text Conditioning Works in Diffusion
The CLIP text encoder converts the text prompt to a sequence of token embeddings (e.g., shape: 77 tokens × 768 dimensions). These embeddings are fed to the U-Net via cross-attention: at each resolution level, the spatial features (from the image) are the queries, and the text embeddings are the keys and values. The network learns to attend to the relevant text tokens while processing each spatial region. This allows the network to generate "fur texture" in positions where the text tokens for "cat" are relevant, and "brushstroke texture" where "Van Gogh" is relevant.
The text prompt is embedded by a CLIP text encoder, then injected as conditioning into the U-Net's cross-attention layers at every denoising step — not just the first. As the reverse process runs from t=T down to t=0, the same conditioning vector keeps steering each step's noise prediction toward the prompt, so structure that matches "cat" and "Van Gogh" gradually emerges from pure noise.
from transformers import CLIPTextModel, CLIPTokenizer
import torch
# Load CLIP text encoder (used inside Stable Diffusion)
tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14")
text_encoder = CLIPTextModel.from_pretrained("openai/clip-vit-large-patch14")
text_encoder.eval()
prompt = "a painting of a cat in the style of Van Gogh, vibrant colors"
# Tokenize and encode
tokens = tokenizer(
prompt,
max_length=77, # CLIP max sequence length
padding="max_length",
truncation=True,
return_tensors="pt"
)
with torch.no_grad():
text_embeddings = text_encoder(tokens.input_ids)[0]
print(f"Tokenized input shape: {tokens.input_ids.shape}") # (1, 77)
print(f"Text embedding shape: {text_embeddings.shape}") # (1, 77, 768)
print(f"This is the conditioning signal injected into the U-Net at every attention layer")
# For CFG we also need the unconditional embedding (empty string)
uncond_tokens = tokenizer(
"",
max_length=77,
padding="max_length",
return_tensors="pt"
)
with torch.no_grad():
uncond_embeddings = text_encoder(uncond_tokens.input_ids)[0]
# Stack them: [uncond, cond] for efficient batched CFG
combined = torch.cat([uncond_embeddings, text_embeddings]) # (2, 77, 768)
print(f"Combined CFG embeddings: {combined.shape}") # (2, 77, 768)7 Latent Diffusion Models: How Stable Diffusion Works
Diffusing directly in pixel space is expensive. A 512×512 RGB image has 786,432 values. Running 1000 denoising steps on this 786k-dimensional space requires enormous compute. Latent Diffusion Models (Rombach et al., 2022) solve this by moving the diffusion process to a compressed latent space.
The VAE Bottleneck
A Variational Autoencoder (VAE) compresses images to a compact latent representation. For Stable Diffusion's VAE: a 512×512×3 image is compressed to a 64×64×4 latent (a ~192× compression ratio). The VAE decoder reconstructs the image from the latent with high perceptual quality — the bottleneck forces the latent to capture the most important visual information.
Stable Diffusion Architecture
The complete Stable Diffusion pipeline: (1) Encode image to latent with VAE encoder (or start from noise in latent space for text-to-image). (2) Run DDIM denoising in latent space with the text-conditioned U-Net (50 steps instead of 1000). (3) Decode the final latent to an image with VAE decoder. The U-Net operates on 64×64×4 instead of 512×512×3 — ~192× fewer values per step, dramatically reducing memory and compute.
# Stable Diffusion inference in ~15 lines
# pip install diffusers accelerate
from diffusers import StableDiffusionPipeline
import torch
# Load pipeline (downloads weights ~4GB on first run)
pipe = StableDiffusionPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
torch_dtype=torch.float16, # fp16 for speed/memory
)
pipe = pipe.to("cuda") # move to GPU
# Generate an image
prompt = "a majestic mountain landscape at sunset, photorealistic, 4k, detailed"
result = pipe(
prompt,
num_inference_steps=50, # DDIM steps (50 is good quality/speed balance)
guidance_scale=7.5, # CFG strength
height=512,
width=512,
generator=torch.Generator("cuda").manual_seed(42), # for reproducibility
)
image = result.images[0]
image.save("generated.png")
print(f"Generated image saved: {image.size}")
# Stable Diffusion XL (SDXL) — higher quality, larger model
from diffusers import StableDiffusionXLPipeline
pipe_xl = StableDiffusionXLPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
torch_dtype=torch.float16,
use_safetensors=True,
)
pipe_xl.to("cuda")
image_xl = pipe_xl(prompt, num_inference_steps=30).images[0]
image_xl.save("generated_xl.png")
# SDXL generates at 1024x1024 by default — 4x more pixels than SD 1.5Before Stable Diffusion (released publicly in August 2022), diffusion models required specialized hardware and thousands of dollars of cloud compute. DALL·E 2 was invite-only. The latent diffusion innovation made a high-quality diffusion model small enough to run on a consumer GPU with 8GB VRAM. Within months, Stable Diffusion had been deployed millions of times, fine-tuned thousands of times, and integrated into products used by tens of millions of people. Open-source AI art generation went from zero to ubiquitous in under a year.
8 Control and Inpainting
Pure text-to-image generation produces great results but gives limited spatial control. Several techniques allow precise control over the spatial layout and content of generated images.
Image-to-Image (img2img)
Start from an existing real image, add noise to a certain level (e.g., t=700 out of 1000 — adding significant but not total noise), then denoise with text guidance. The initial image structure is preserved by the early denoising steps, while the text prompt guides the stylistic changes. This lets you take a photograph and paint it in the style of Monet, or take a rough sketch and turn it into a detailed illustration.
Inpainting
Given an image and a binary mask, diffuse only the masked (erased) region while keeping the unmasked pixels unchanged. This enables seamless removal of objects (remove the person from the background), background replacement (keep the subject, replace the background), or object insertion (add a specific object into a scene). The model must generate content that blends realistically with the surrounding unmasked pixels.
ControlNet: Spatial Conditioning
ControlNet (Zhang et al., 2023) adds additional input channels to the U-Net, allowing the model to condition on spatial control signals: edge maps (Canny edges), depth maps, human pose keypoints (OpenPose), semantic segmentation maps, or even another image. You can draw a stick figure and ControlNet generates a photorealistic person in that exact pose. You can provide a depth map and generate an image with that exact 3D structure.
from diffusers import StableDiffusionImg2ImgPipeline, StableDiffusionInpaintPipeline
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
from PIL import Image
import torch
device = "cuda" if torch.cuda.is_available() else "cpu"
# ── img2img ──
img2img_pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
torch_dtype=torch.float16,
).to(device)
init_image = Image.open("photo.jpg").resize((512, 512))
result = img2img_pipe(
prompt="a painting in the style of Monet, impressionist, soft brushstrokes",
image=init_image,
strength=0.7, # 0=no change, 1=ignore original, 0.7=significant stylization
guidance_scale=7.5,
)
result.images[0].save("img2img_output.jpg")
# ── Inpainting ──
inpaint_pipe = StableDiffusionInpaintPipeline.from_pretrained(
"runwayml/stable-diffusion-inpainting",
torch_dtype=torch.float16,
).to(device)
original = Image.open("beach.jpg").resize((512, 512))
# mask: white pixels = area to regenerate; black = keep original
mask = Image.open("mask.png").resize((512, 512))
result = inpaint_pipe(
prompt="a red sports car parked on the beach",
image=original,
mask_image=mask,
guidance_scale=7.5,
)
result.images[0].save("inpainted.jpg")
# ── ControlNet (Canny edges) ──
controlnet = ControlNetModel.from_pretrained(
"lllyasviel/sd-controlnet-canny",
torch_dtype=torch.float16,
)
cnet_pipe = StableDiffusionControlNetPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
controlnet=controlnet,
torch_dtype=torch.float16,
).to(device)
import cv2, numpy as np
from PIL import Image as PILImage
# Extract Canny edges from a reference image
ref_img = cv2.imread("reference.jpg")
edges = cv2.Canny(ref_img, 100, 200)
edge_img = PILImage.fromarray(edges)
result = cnet_pipe(
prompt="a photograph of a modern building, architecture photography, HDR",
image=edge_img, # ControlNet uses edge map as spatial condition
num_inference_steps=20,
guidance_scale=9.0,
)
result.images[0].save("controlnet_output.jpg")
print("ControlNet: generate image with exact spatial structure from edge map")Text-to-image: total creative freedom, no spatial control. img2img: preserve rough layout, change style (strength=0.5–0.8). Inpainting: change specific region, preserve rest. ControlNet: precise spatial control via structural signal. In practice: start with img2img for stylization, use ControlNet when you need a specific composition or pose, use inpainting for removing/adding objects to existing images. These techniques can be combined: ControlNet + inpainting allows controlled object insertion with spatial constraints.
Real-World Spotlight: Diffusion Models in Science and Creative Industries
Creative Industry Applications
Stable Diffusion and SDXL have become standard tools in concept art, game development, and advertising. Adobe Firefly (integrated into Photoshop) uses a proprietary diffusion model trained on licensed content. Midjourney uses a transformer-based diffusion architecture and has over 15 million active users. DALL·E 3 (integrated into ChatGPT) generates images from conversational descriptions, with dramatically better text rendering than previous models.
Scientific Applications: AlphaFold3 and Drug Discovery
AlphaFold3 (DeepMind, 2024) predicts 3D protein structures using a diffusion process over atomic coordinates. Instead of noising pixel values, it noises the 3D positions of atoms and learns to denoise them conditioned on the amino acid sequence. This enabled predictions of protein-DNA, protein-RNA, and protein-small molecule complexes with unprecedented accuracy — directly enabling drug discovery. The core insight: diffusion models generalize far beyond images to any structured data that can be represented as a vector (molecules, protein structures, audio waveforms, videos).
# Video generation with Stable Video Diffusion
# pip install diffusers[torch]
from diffusers import StableVideoDiffusionPipeline
from diffusers.utils import load_image, export_to_video
import torch
pipe = StableVideoDiffusionPipeline.from_pretrained(
"stabilityai/stable-video-diffusion-img2vid-xt",
torch_dtype=torch.float16,
variant="fp16",
)
pipe.enable_model_cpu_offload()
# Generate a video from an image
image = load_image("landscape.jpg")
image = image.resize((1024, 576))
frames = pipe(
image,
num_frames=25,
decode_chunk_size=8,
generator=torch.manual_seed(42),
).frames[0]
export_to_video(frames, "animation.mp4", fps=7)
print("Stable Video Diffusion: single image → 25-frame video")
print("Sora (OpenAI): video diffusion at 1080p, 60s clips — the current frontier")Diffusion models raise important ethical questions: (1) Training data copyright — Stability AI faces lawsuits over training on copyrighted images without consent. (2) Image authenticity — it is now trivial to generate photorealistic images of events that never happened. (3) Deepfakes — generating photorealistic images or videos of specific real people. (4) Consent — generating images in specific real artists' styles. The field is actively developing technical mitigations (concept erasure, watermarking) and regulatory frameworks (EU AI Act, US Executive Order on AI). Understanding these issues is as important as understanding the technology.
✍️ Practice Exercises
- Implement the
NoiseScheduleclass from Section 2. Visualize the noise schedule by plotting ᾱ_t (signal strength) against timestep for both a linear schedule and a cosine schedule (ᾱ_t = cos²(π·t/(2T))). Which schedule maintains signal strength longer in early timesteps? - Implement a minimal DDPM training loop for MNIST digit generation. Use a simple U-Net with timestep embeddings. Train for 50 epochs on MNIST. Sample 16 images using the reverse process and display them.
- Using the Hugging Face
diffuserslibrary, generate 5 images with the same prompt at different guidance scales (1.0, 3.0, 7.5, 12.0, 20.0). Display them side by side and describe how the guidance scale affects visual quality and diversity. - Implement img2img with Stable Diffusion: take a real photograph, apply img2img at three different strength values (0.3, 0.6, 0.9), and compare the results. At what strength does the generated image lose too much of the original structure?
▶ Show Solution (Exercise 1 — Cosine vs Linear Schedule)
import torch
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
T = 1000
# Linear schedule
betas_linear = torch.linspace(1e-4, 0.02, T)
alpha_bars_linear = torch.cumprod(1 - betas_linear, dim=0)
# Cosine schedule (Nichol & Dhariwal 2021)
s = 0.008
t = torch.linspace(0, T, T + 1)
f = torch.cos((t / T + s) / (1 + s) * torch.pi / 2) ** 2
alpha_bars_cosine = (f / f[0])[1:] # normalize and trim
timesteps = torch.arange(T)
plt.figure(figsize=(10, 5))
plt.plot(timesteps, alpha_bars_linear, label='Linear schedule', color='blue')
plt.plot(timesteps, alpha_bars_cosine, label='Cosine schedule', color='orange')
plt.xlabel('Timestep t')
plt.ylabel('ᾱ_t (signal strength)')
plt.title('Noise Schedules: Signal Strength vs Timestep')
plt.legend()
plt.grid(True, alpha=0.3)
plt.savefig('noise_schedules.png', dpi=150, bbox_inches='tight')
print("Linear: signal decays roughly linearly — drops quickly in first 500 steps")
print("Cosine: signal decays slowly at first, then accelerates — more gradual and")
print(" provides more useful training signal at small noise levels")📚 Primary Source for This Lesson
Ho, Jain & Abbeel (2020) — "Denoising Diffusion Probabilistic Models" (DDPM)
The paper that established the modern diffusion training procedure covered in this lesson. For the latent-space version powering Stable Diffusion, see Rombach et al. (2022) "High-Resolution Image Synthesis with Latent Diffusion Models."