🎯 What You'll Learn
- Why full fine-tuning of 7B+ parameter models is prohibitively expensive for most practitioners
- The mathematical insight behind LoRA: why weight updates during fine-tuning are inherently low-rank
- LoRA's mechanics: the A and B matrices, rank r, the alpha scaling factor, and zero initialization
- Which layers to apply LoRA to and why
- QLoRA: 4-bit quantization + LoRA adapters = a 7B model on 6GB of GPU memory
- Use the Hugging Face
peftlibrary to configure, train, save, and load LoRA adapters - Prefix tuning and prompt tuning as alternative PEFT approaches
- The complete fine-tuning workflow with alpaca-format data
- When to use LoRA fine-tuning vs RAG for domain adaptation
A 7-billion parameter model like Llama 3 has 7 billion weights. Full fine-tuning updates all of them — requiring 14GB+ of GPU memory just to store the weights in fp16, plus gradients and optimizer states pushing past 100GB total. That's 4–8 high-end GPUs. LoRA is a brilliant hack: instead of updating the full weight matrix W, you approximate the update as ΔW = A × B where A and B are tiny low-rank matrices. Fine-tuning a 7B model with LoRA might update only 4M parameters instead of 7B — that's 0.06% of the model — fitting on a single consumer GPU. Same task performance. 99.9% fewer trainable parameters.
1 The Fine-Tuning Cost Problem
Before LoRA, fine-tuning a large language model was reserved for organisations with dedicated ML infrastructure. Let's calculate exactly why — because understanding the memory breakdown tells you what LoRA needs to solve.
Memory Breakdown for Full Fine-tuning a 7B Model
For a model with N parameters, you need memory for:
- Model weights (fp16): 2 bytes × 7B = 14 GB
- Gradients (fp16): another 2 bytes × 7B = 14 GB (one gradient per parameter)
- Adam optimizer states: Adam stores a first moment (moving average of gradients) and a second moment (moving average of squared gradients) — both in fp32 = 4 bytes each × 7B × 2 states = 56 GB
- Activations (for backprop): depends on batch size and sequence length, typically 10–30 GB
Total: ~100 GB+ for a 7B model with batch size 4. Even an A100 80GB GPU isn't enough without techniques like gradient checkpointing.
The Business Reality
An A100 80GB cloud instance costs ~$3/hour. Gradient checkpointing and DeepSpeed can get a 7B fine-tune to fit on 4× A100s (~$12/hour). A typical fine-tuning run takes 3–10 hours. Cost: $36–$120 just for compute. With LoRA + 4-bit quantization (QLoRA), the same fine-tune fits on a single RTX 3090 (24GB) — costing ~$0.30/hour on cloud or nothing if you own the GPU.
def estimate_finetuning_memory(
num_params_billions: float,
method: str = 'full',
dtype: str = 'fp16'
) -> dict:
"""Estimate GPU memory requirements for fine-tuning."""
N = num_params_billions * 1e9
bytes_per_param = {'fp16': 2, 'fp32': 4, 'int8': 1, 'int4': 0.5}[dtype]
if method == 'full':
weights = N * bytes_per_param
gradients = N * bytes_per_param # same dtype as weights
adam_state = N * 4 * 2 # fp32, 2 moments
activations = N * 2 * 0.3 # rough estimate
total = weights + gradients + adam_state + activations
elif method == 'lora':
lora_fraction = 0.001 # ~0.1% of params are trainable
weights = N * bytes_per_param # base model still needed
gradients = N * lora_fraction * 4 # only LoRA params need gradients
adam_state = N * lora_fraction * 8 # fp32 Adam for LoRA params
activations = N * 2 * 0.3
total = weights + gradients + adam_state + activations
elif method == 'qlora':
lora_fraction = 0.001
weights = N * 0.5 # int4 quantized base model
gradients = N * lora_fraction * 2 # fp16 LoRA gradients
adam_state = N * lora_fraction * 8 # fp32 Adam for LoRA params
activations = N * 0.5 * 0.3 # reduced by quantization
total = weights + gradients + adam_state + activations
return {
'weights_GB': weights / 1e9,
'gradients_GB': gradients / 1e9,
'optimizer_GB': adam_state / 1e9,
'activations_GB': activations / 1e9,
'total_GB': total / 1e9
}
print(f"{'Method':<10} {'Weights':>10} {'Grads':>10} {'Optim':>10} {'Activations':>12} {'TOTAL':>10}")
print("─" * 65)
for method in ['full', 'lora', 'qlora']:
m = estimate_finetuning_memory(7.0, method=method)
print(f"{method:<10} {m['weights_GB']:>8.1f}GB {m['gradients_GB']:>8.1f}GB "
f"{m['optimizer_GB']:>8.1f}GB {m['activations_GB']:>10.1f}GB {m['total_GB']:>8.1f}GB")
print("\nGPU memory reference:")
print(" RTX 3080/3090: 10–24 GB (consumer, ~$800)")
print(" RTX 4090: 24 GB (prosumer, ~$1,600)")
print(" A100 40GB: 40 GB (data center, ~$10,000)")
print(" A100 80GB: 80 GB (data center, ~$15,000)")2 The Intrinsic Dimensionality Insight
Why does LoRA work at all? The justification comes from research into the intrinsic dimensionality of fine-tuning.
What Actually Changes During Fine-tuning?
When you take a pre-trained BERT or GPT and fine-tune it on a downstream task, what happens to the weights? The pre-trained model already "knows" language. The fine-tuning step is teaching it something new — usually a much simpler mapping than "understand all of English." The weight changes ΔW = W_fine_tuned − W_pre_trained encode this new task-specific information.
Li et al. (2018) showed empirically that neural networks can be fine-tuned effectively within a very low-dimensional subspace. Even though the model has millions of parameters, the fine-tuning essentially explores a much smaller effective parameter space. The intuition: you are not learning language from scratch — you are making small, structured adjustments to an already-competent system.
The Matrix Rank Perspective
Consider a weight matrix W of size 768×768 (as in BERT-base). Its rank could be up to 768. But the weight CHANGE ΔW from pre-training to fine-tuning has empirically been shown to be low-rank — you can approximate it very well with a rank-4 or rank-8 matrix. This is the key mathematical justification for LoRA.
Think of it this way: imagine the pre-trained model sits in a high-dimensional space. The fine-tuning task only requires movement in a small subspace of that space — like moving around a plane in 3D space (2D subspace) rather than needing all 3 dimensions. LoRA explicitly models this low-rank structure.
import numpy as np
import torch
# Demonstrate low-rank structure of weight changes empirically
# We'll show that random high-rank matrices are compressible with SVD
# while "task-specific" changes concentrate in a few singular vectors
np.random.seed(42)
d = 768 # BERT-hidden dim
# Simulate a "full fine-tuning" weight update (random = not low-rank)
delta_W_random = np.random.randn(d, d) * 0.01
# Compute SVD and check how many singular values are needed
U, S, Vt = np.linalg.svd(delta_W_random, full_matrices=False)
# S is sorted descending — plot cumulative variance
cumvar = np.cumsum(S**2) / np.sum(S**2)
r_90 = np.searchsorted(cumvar, 0.90) + 1
r_99 = np.searchsorted(cumvar, 0.99) + 1
print("Random weight matrix (no structure):")
print(f" Rank for 90% variance: {r_90} out of {d}")
print(f" Rank for 99% variance: {r_99} out of {d}")
# Simulate a task-specific weight update (low-rank by construction)
# In reality, fine-tuning updates have this structure empirically
true_r = 8 # low rank
A_true = np.random.randn(d, true_r) * 0.01
B_true = np.random.randn(true_r, d) * 0.01
delta_W_lowrank = A_true @ B_true # inherently rank-8
U2, S2, Vt2 = np.linalg.svd(delta_W_lowrank, full_matrices=False)
cumvar2 = np.cumsum(S2**2) / np.sum(S2**2)
r_90_2 = np.searchsorted(cumvar2, 0.90) + 1
r_99_2 = np.searchsorted(cumvar2, 0.99) + 1
print("\nLow-rank weight update (task-specific structure):")
print(f" True rank: {true_r}")
print(f" Rank for 90% variance: {r_90_2}")
print(f" Rank for 99% variance: {r_99_2}")
# Reconstruct the low-rank delta_W using only rank-16 approximation
r_approx = 16
delta_W_approx = U2[:, :r_approx] @ np.diag(S2[:r_approx]) @ Vt2[:r_approx, :]
reconstruction_error = np.linalg.norm(delta_W_lowrank - delta_W_approx, 'fro')
original_norm = np.linalg.norm(delta_W_lowrank, 'fro')
print(f"\nReconstruction with rank-{r_approx}: error = {reconstruction_error/original_norm:.6f} (near 0 = perfect)")3 LoRA: Low-Rank Adaptation
LoRA (Hu et al., 2022) operationalises the low-rank insight into a clean, practical algorithm. Here is the complete formulation:
The Core Idea
For a weight matrix W of shape d×k, instead of adding a trainable update ΔW of size d×k (which is expensive), introduce two small matrices:
- A of shape d×r (called the "up-projection")
- B of shape r×k (called the "down-projection")
where r is the rank, typically 4–64, and r << min(d, k). The low-rank update is: ΔW = B × A.
During the forward pass: h = Wx + (BA)x × (α/r), where α is a scaling hyperparameter that controls the magnitude of the LoRA update. Only A and B are updated during training; W is frozen.
LoRA freezes the original weight matrix W and routes a parallel low-rank path through two small trainable matrices, A (d×r) and B (r×d). Their product BA approximates the weight update ΔW, scaled by α/r, and is added to W's output. Only the violet A/B path is trained — W never changes.
Initialization: Why B is Zeros
A is initialized with random Gaussian values (standard kaiming init). B is initialized to all zeros. At the start of training, ΔW = B×A = 0×A = 0 — the model starts exactly at the pre-trained weights, with no random perturbation. This is crucial: if B were random, the model would start in a degraded state, making early training unstable.
Parameter Count
A matrix has d×r parameters. B matrix has r×k parameters. Total LoRA parameters for one layer: d×r + r×k = r(d+k). Compare to the original d×k. For a typical 768×768 BERT layer with r=8: 8×(768+768) = 12,288 parameters vs 768×768 = 589,824. A 48× reduction per layer.
Drag the rank slider below to see this tradeoff live. We fix a square weight matrix of size d×d = 4096×4096 (a typical LLM hidden dimension) and vary the LoRA rank r. LoRA's trainable parameter count is 2 × d × r (matrix A is d×r, matrix B is r×d) — it grows only linearly in r, while full fine-tuning of that matrix always costs the fixed d × d. Watch how far r has to climb before the LoRA bar even begins to approach the full fine-tuning bar:
r = 8 — LoRA trains 65,536 parameters vs 16,777,216 for full fine-tuning (0.39%).
import torch
import torch.nn as nn
class LoRALinear(nn.Module):
"""A linear layer augmented with a LoRA adapter.
The forward pass computes: output = W @ x + (B @ A) @ x * (alpha / rank)
During training, only A and B are updated; W is frozen.
"""
def __init__(self, in_features: int, out_features: int,
rank: int = 8, lora_alpha: float = 16.0, dropout: float = 0.0):
super().__init__()
self.in_features = in_features
self.out_features = out_features
self.rank = rank
self.scaling = lora_alpha / rank # the α/r scaling factor
# Frozen pre-trained weights
self.weight = nn.Parameter(torch.empty(out_features, in_features), requires_grad=False)
self.bias = nn.Parameter(torch.zeros(out_features), requires_grad=False)
nn.init.kaiming_uniform_(self.weight, a=5**0.5)
# Trainable LoRA matrices
self.lora_A = nn.Parameter(torch.empty(rank, in_features)) # down-projection
self.lora_B = nn.Parameter(torch.zeros(out_features, rank)) # up-projection (ZERO INIT)
nn.init.kaiming_uniform_(self.lora_A, a=5**0.5)
# lora_B is already zeros — ΔW = B@A = 0 at initialization ✓
self.dropout = nn.Dropout(dropout) if dropout > 0 else nn.Identity()
def forward(self, x: torch.Tensor) -> torch.Tensor:
# Pretrained component (frozen)
base_out = nn.functional.linear(x, self.weight, self.bias)
# LoRA component (trainable)
lora_out = nn.functional.linear(
self.dropout(x),
self.lora_B @ self.lora_A # ΔW = B @ A, shape: (out, in)
) * self.scaling
return base_out + lora_out
def merge(self) -> nn.Linear:
"""Merge LoRA weights into the base weight for efficient inference."""
merged = nn.Linear(self.in_features, self.out_features)
merged.weight.data = self.weight + (self.lora_B @ self.lora_A) * self.scaling
merged.bias.data = self.bias
return merged
# Demonstrate parameter counting
d_model = 768
lora_layer = LoRALinear(d_model, d_model, rank=8, lora_alpha=16)
total_params = sum(p.numel() for p in lora_layer.parameters())
frozen_params = sum(p.numel() for p in lora_layer.parameters() if not p.requires_grad)
trainable_params = sum(p.numel() for p in lora_layer.parameters() if p.requires_grad)
print(f"LoRA layer ({d_model}×{d_model}, rank=8):")
print(f" Total parameters: {total_params:,}")
print(f" Frozen (base): {frozen_params:,}")
print(f" Trainable (LoRA): {trainable_params:,}")
print(f" Trainable fraction: {trainable_params/total_params*100:.2f}%")
# Test that ΔW = 0 at initialization
delta_W = lora_layer.lora_B @ lora_layer.lora_A
print(f"\n ΔW = B@A at init: {delta_W.abs().sum().item():.6f} (should be 0.0)")
# Test forward pass
x = torch.randn(4, 20, d_model) # batch=4, seq=20
out = lora_layer(x)
print(f" Input shape: {x.shape} → Output shape: {out.shape}")Rank r controls the expressivity of the adapter. Common values: r=4 (very efficient, works for simple tasks), r=8 (good default), r=16 (better for complex tasks), r=64 (approaching full fine-tuning expressivity). The scaling α should generally be set to 2×r (i.e., alpha=16 for rank=8). Increasing r increases trainable parameters and memory roughly linearly — there are diminishing returns above r=32 for most fine-tuning tasks.
4 Which Layers to Apply LoRA To
A Transformer has many weight matrices: Q, K, V, output projection (Wo) in each attention block, plus the two FFN weight matrices (W1 and W2). You don't have to LoRA all of them. The choice is a tradeoff between expressivity (more layers = more adaptable) and efficiency (more layers = more trainable parameters).
Common Choices
Minimum (fastest): only Q and V projection matrices. This covers the attention-mechanism-level adaptation. Works well for most NLP classification and generation tasks. In PEFT terms: target_modules=['q_proj', 'v_proj'] for Llama-style models or target_modules=['query', 'value'] for BERT.
Standard: all four attention projections (Q, K, V, Wo). This gives more expressivity for the attention mechanism. Roughly doubles the LoRA parameters vs. Q+V only.
Maximum (most expressive): all attention projections + FFN weight matrices. Some papers (e.g., Llama-3 fine-tuning guides) include the FFN layers for better performance on complex generation tasks. Triples or quadruples parameter count vs. Q+V.
import torch
import torch.nn as nn
# Demonstrate applying LoRA to specific layers by name
def apply_lora_to_model(model: nn.Module, target_modules: list, rank: int = 8) -> nn.Module:
"""Replace specified linear layers with LoRALinear equivalents."""
for name, module in model.named_modules():
parent_name = '.'.join(name.split('.')[:-1])
child_name = name.split('.')[-1]
if isinstance(module, nn.Linear) and child_name in target_modules:
parent = model
for part in parent_name.split('.'):
if part:
parent = getattr(parent, part)
# Replace with LoRA version
lora_module = LoRALinear(
module.in_features, module.out_features,
rank=rank, lora_alpha=rank * 2
)
# Copy pre-trained weights into frozen part
lora_module.weight.data = module.weight.data.clone()
if module.bias is not None:
lora_module.bias.data = module.bias.data.clone()
setattr(parent, child_name, lora_module)
return model
# Example: count parameters for different target_modules choices on a small model
from transformers import GPT2Model, GPT2Config
def count_lora_params_for_targets(target_modules):
model = GPT2Model(GPT2Config(n_layer=12, n_head=12, n_embd=768))
total = sum(p.numel() for p in model.parameters())
# Count params in target modules
trainable = 0
for name, module in model.named_modules():
if isinstance(module, nn.Linear):
child_name = name.split('.')[-1]
if child_name in target_modules:
rank = 8
trainable += rank * (module.in_features + module.out_features)
return total, trainable
scenarios = {
'Q, V only': ['c_attn'], # GPT-2 combines Q,K,V in one matrix
'All attention': ['c_attn', 'c_proj'],
'Attention + FFN': ['c_attn', 'c_proj', 'c_fc', 'c_proj'],
}
print(f"\n{'Target Modules':<25} {'Total Params':>14} {'LoRA Trainable':>16} {'%':>6}")
print("─" * 66)
for scenario_name, targets in scenarios.items():
total, trainable = count_lora_params_for_targets(targets)
print(f"{scenario_name:<25} {total:>12,} {trainable:>14,} {trainable/total*100:>5.2f}%")5 QLoRA: Fitting 7B Models on One GPU
LoRA halves trainable parameters but the frozen base model still occupies its full fp16 memory. For a 7B model, that's still 14GB — exceeding most consumer GPUs. QLoRA (Dettmers et al., 2023) adds one more trick: quantize the base model to 4-bit precision.
4-bit NormalFloat (NF4) Quantization
Standard 4-bit quantization would just uniformly discretize the weight range — losing precision in dense regions of the weight distribution. NF4 is smarter: it allocates the 16 quantization levels (4 bits = 16 values) so they are evenly spaced in terms of probability under a normal distribution, which is the typical distribution of neural network weights. This preserves more information in the dense central region of the weight distribution.
The result: 4-bit NF4 weights use 0.5 bytes per parameter instead of 2 bytes (fp16) — a 4× memory reduction. A 7B model that needed 14GB now needs 3.5GB. Add LoRA adapters in fp16 (+~200MB) and you're fine-tuning a 7B model in under 6GB total.
Double Quantization
QLoRA goes one step further: the quantization constants themselves (the scale and zero-point needed to dequantize each block) are also quantized from fp32 to fp8. This saves an additional ~0.4 bits per parameter. At 7B parameters, this saves ~400MB — not huge, but every MB counts at this scale.
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
# ── QLoRA: Load model in 4-bit ──
bnb_config = BitsAndBytesConfig(
load_in_4bit=True, # use 4-bit NF4 quantization
bnb_4bit_use_double_quant=True, # double quantize the quantization constants
bnb_4bit_quant_type='nf4', # NormalFloat4 (better than linear int4)
bnb_4bit_compute_dtype=torch.bfloat16, # compute in bfloat16 after dequantization
)
# Load a model in 4-bit (requires bitsandbytes library)
# Using GPT-2 here for demo; for real QLoRA use Llama/Mistral
model_name = 'gpt2' # replace with 'meta-llama/Llama-3.2-3B' for real use
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map='auto', # automatically distribute across available GPUs
)
# Memory comparison (illustrative for 7B models)
print("Memory estimates for a 7B parameter model:")
print(f" fp32 full fine-tuning: ~112 GB")
print(f" fp16 full fine-tuning: ~56 GB")
print(f" fp16 + LoRA (r=16): ~18 GB")
print(f" int8 quantization + LR: ~9 GB")
print(f" QLoRA (nf4 + r=16): ~6 GB ← fits on RTX 3090/4090!")
# Inspect quantized model
print(f"\nModel dtype: {model.dtype}")
print(f"Model device: {model.device}")
for name, module in list(model.named_modules())[:5]:
print(f" {name}: {type(module).__name__}")6 Hugging Face PEFT Library: Hands-on
The peft library from Hugging Face provides a clean API for all PEFT methods — LoRA, prefix tuning, prompt tuning, and more. It integrates seamlessly with the Hugging Face Trainer API.
from peft import (LoraConfig, TaskType, get_peft_model,
PeftModel, prepare_model_for_kbit_training)
from transformers import (AutoModelForCausalLM, AutoTokenizer,
BitsAndBytesConfig, TrainingArguments, Trainer,
DataCollatorForLanguageModeling)
from datasets import load_dataset
import torch
# ── Step 1: Load base model (with 4-bit quantization for QLoRA) ──
model_name = 'gpt2' # use 'meta-llama/Llama-3.2-3B-Instruct' in production
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type='nf4',
bnb_4bit_compute_dtype=torch.bfloat16,
)
# For GPT-2 demo without bitsandbytes, skip the bnb_config:
base_model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
# ── Step 2: Prepare model for k-bit training (QLoRA specific) ──
# This enables gradient checkpointing and ensures proper dtype casting
# base_model = prepare_model_for_kbit_training(base_model)
# ── Step 3: Configure LoRA ──
lora_config = LoraConfig(
r=16, # rank — number of dimensions in low-rank matrices
lora_alpha=32, # alpha = 2*r is a safe default
target_modules=['c_attn'], # GPT-2 uses c_attn for Q,K,V
lora_dropout=0.05, # dropout on LoRA adapter
bias='none', # don't add LoRA to bias terms
task_type=TaskType.CAUSAL_LM, # we are doing causal language modeling
)
# ── Step 4: Wrap the base model with LoRA ──
model = get_peft_model(base_model, lora_config)
# This is the crucial line — prints trainable vs total parameters
model.print_trainable_parameters()
# ── Step 5: Verify the architecture ──
print("\nFirst transformer block attention layer:")
print(type(model.base_model.model.transformer.h[0].attn))
# Should show 'lora.Linear' instead of standard 'Conv1D'
# ── Step 6: Prepare dataset (Alpaca format for instruction tuning) ──
def format_alpaca(example: dict) -> str:
"""Format a sample in Alpaca instruction format."""
if example.get('input', ''):
return (f"### Instruction:\n{example['instruction']}\n\n"
f"### Input:\n{example['input']}\n\n"
f"### Response:\n{example['output']}")
else:
return (f"### Instruction:\n{example['instruction']}\n\n"
f"### Response:\n{example['output']}")
# Use a small subset for demo
dataset = load_dataset('tatsu-lab/alpaca', split='train[:500]')
dataset = dataset.map(lambda x: {'text': format_alpaca(x)}, remove_columns=dataset.column_names)
def tokenize(batch):
return tokenizer(batch['text'], truncation=True, max_length=512,
padding='max_length')
tokenized = dataset.map(tokenize, batched=True, remove_columns=['text'])
tokenized.set_format('torch')
split = tokenized.train_test_split(test_size=0.1)
# ── Step 7: Train ──
training_args = TrainingArguments(
output_dir='./lora_gpt2',
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=4, # effective batch size = 4×4 = 16
learning_rate=2e-4, # higher LR for LoRA (only tiny fraction of params)
warmup_ratio=0.05,
weight_decay=0.001,
evaluation_strategy='epoch',
save_strategy='epoch',
fp16=torch.cuda.is_available(),
report_to='none',
logging_steps=20,
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=split['train'],
eval_dataset=split['test'],
data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),
)
trainer.train()
# ── Step 8: Save only the adapter (NOT the full model) ──
model.save_pretrained('./lora_adapter')
tokenizer.save_pretrained('./lora_adapter')
import os
adapter_size = sum(
os.path.getsize(os.path.join('./lora_adapter', f))
for f in os.listdir('./lora_adapter')
) / 1e6
print(f"\nAdapter saved! Size: {adapter_size:.1f} MB (vs ~500 MB for full GPT-2)")
# ── Step 9: Load and use the adapter ──
base = AutoModelForCausalLM.from_pretrained('gpt2')
peft_model = PeftModel.from_pretrained(base, './lora_adapter')
peft_model.eval()
inputs = tokenizer("### Instruction:\nWrite a haiku about Python programming.\n\n### Response:\n",
return_tensors='pt')
with torch.no_grad():
output = peft_model.generate(inputs['input_ids'], max_new_tokens=60,
do_sample=True, top_p=0.9, temperature=0.8,
pad_token_id=tokenizer.eos_token_id)
print("\nFine-tuned response:")
print(tokenizer.decode(output[0], skip_special_tokens=True))7 Other PEFT Methods: Prefix Tuning & Prompt Tuning
LoRA is not the only PEFT approach. Two other methods are worth understanding — they take a completely different approach and have complementary strengths.
Prefix Tuning (Li & Liang, 2021)
Rather than modifying the weight matrices, prefix tuning prepends K learnable "virtual token" vectors to the input of every Transformer layer. These prefix vectors are not actual tokens from the vocabulary — they're free-floating vectors in the activation space that the model learns to attend to appropriately for each task. The model's weights are completely frozen; only the prefix vectors are trained.
Intuition: instead of changing the model's "character" (weight matrices), you're adding a learnable "context" at every layer that steers the model's attention. The prefix acts like a persistent prompt that the model always has access to.
Prompt Tuning (Lester et al., 2021)
Even simpler than prefix tuning: only add learnable tokens to the input embedding layer, not every layer. Prepend P learnable embedding vectors before the actual input embeddings. Only these P×d_model parameters are trained. The advantage: even fewer parameters than LoRA. The catch: performance is scale-dependent — this works well for models above 10B parameters but struggles with smaller models. For GPT-2 scale, LoRA consistently outperforms prompt tuning.
from peft import PrefixTuningConfig, PromptTuningConfig, PromptTuningInit
# ── Prefix Tuning ──
prefix_config = PrefixTuningConfig(
task_type=TaskType.CAUSAL_LM,
num_virtual_tokens=20, # prepend 20 virtual tokens per layer
prefix_projection=True, # use an MLP to project prefix vectors
encoder_hidden_size=512, # MLP hidden size
)
# model = get_peft_model(base_model, prefix_config)
# model.print_trainable_parameters()
# Expected: ~1.2M params (20 tokens × 768 dim × 12 layers × 2 for MLP) ≈ 0.96%
# ── Prompt Tuning ──
prompt_config = PromptTuningConfig(
task_type=TaskType.CAUSAL_LM,
num_virtual_tokens=20, # 20 learnable prefix tokens
tokenizer_name_or_path='gpt2',
prompt_tuning_init=PromptTuningInit.TEXT, # initialize from real text
prompt_tuning_init_text="Generate text following the instruction:",
)
# model = get_peft_model(base_model, prompt_config)
# model.print_trainable_parameters()
# Expected: 20 × 768 = 15,360 params = 0.012% of GPT-2
# Comparison table
print("── PEFT Method Comparison ──")
print(f"\n{'Method':<18} {'Where applied':<30} {'Typical Params':<18} {'Best For'}")
print("─" * 90)
methods = [
("LoRA", "Weight matrices (Q, V, ...)", "0.06% – 1%", "Most tasks; best accuracy/efficiency"),
("Prefix Tuning", "Prefix at every layer", "0.1% – 1%", "Generation; sequence-to-sequence"),
("Prompt Tuning", "Input embeddings only", "<0.01%", "Very large models (>10B); few-shot style"),
("Adapter", "Bottleneck layers between FFN", "0.5% – 5%", "Multi-task fine-tuning"),
("Full FT", "All weight matrices", "100%", "Maximum accuracy; data-rich settings"),
]
for name, where, params, use in methods:
print(f"{name:<18} {where:<30} {params:<18} {use}")8 Practical Fine-tuning Workflow with LoRA
Here is the complete practical workflow for instruction-tuning an LLM with LoRA. This is the playbook used by the community for fine-tuning Llama, Mistral, and other open-source models on custom tasks.
Dataset Format: Alpaca vs ShareGPT
Alpaca format (single-turn instruction following):
alpaca_sample = {
"instruction": "Translate the following English text to French.",
"input": "The weather is beautiful today.",
"output": "Le temps est magnifique aujourd'hui."
}
def format_alpaca_prompt(sample: dict) -> str:
"""Convert an Alpaca-format sample to a training string."""
if sample.get('input', '').strip():
return (f"Below is an instruction that describes a task, paired with an input "
f"that provides further context. Write a response.\n\n"
f"### Instruction:\n{sample['instruction']}\n\n"
f"### Input:\n{sample['input']}\n\n"
f"### Response:\n{sample['output']}")
else:
return (f"Below is an instruction that describes a task. Write a response.\n\n"
f"### Instruction:\n{sample['instruction']}\n\n"
f"### Response:\n{sample['output']}")
# ShareGPT format (multi-turn conversations)
sharegpt_sample = {
"conversations": [
{"from": "human", "value": "Explain what a neural network is."},
{"from": "gpt", "value": "A neural network is a computational model..."},
{"from": "human", "value": "How does backpropagation work?"},
{"from": "gpt", "value": "Backpropagation is the algorithm used to..."},
]
}
def format_chatml(conversation: list) -> str:
"""Convert a conversation to ChatML format (used by Llama-3, Mistral)."""
result = ""
for turn in conversation:
role = "user" if turn['from'] == 'human' else "assistant"
result += f"<|im_start|>{role}\n{turn['value']}<|im_end|>\n"
return resultTraining Hyperparameters for LoRA
LoRA fine-tuning uses different hyperparameters than full fine-tuning because you're only training a tiny fraction of parameters. The key differences:
- Learning rate: 2e-4 to 3e-4 (vs 2e-5 for full FT). LoRA parameters start at zero and need a larger step size to learn quickly.
- Epochs: 3–5. Overfitting happens fast with small datasets — monitor validation loss carefully.
- Gradient accumulation: use
gradient_accumulation_steps=4or higher if your GPU can't fit a large batch. Effective batch size = per_device_batch × accumulation_steps. - Warmup: 5–10% of total steps. LoRA parameters need a gentler warmup than full FT.
from transformers import TrainingArguments
# Production-grade LoRA training arguments
production_args = TrainingArguments(
output_dir='./lora_output',
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=4, # effective batch = 16
learning_rate=2e-4, # higher than full FT
lr_scheduler_type='cosine', # cosine annealing
warmup_ratio=0.05,
weight_decay=0.001,
max_grad_norm=0.3, # gradient clipping for stability
evaluation_strategy='steps',
eval_steps=100,
save_strategy='steps',
save_steps=100,
load_best_model_at_end=True,
fp16=False,
bf16=True, # bfloat16 is preferred for modern GPUs
dataloader_num_workers=4,
group_by_length=True, # batch similar-length sequences (reduces padding)
report_to='none',
logging_steps=10,
optim='paged_adamw_8bit', # quantized AdamW — saves optimizer memory
)
# Merging LoRA weights into base model for deployment
def merge_and_save(base_model_name: str, lora_adapter_path: str, output_path: str):
"""Merge LoRA adapter into base model and save the full model."""
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
print(f"Loading base model: {base_model_name}")
base = AutoModelForCausalLM.from_pretrained(base_model_name, torch_dtype=torch.float16)
print(f"Loading LoRA adapter: {lora_adapter_path}")
peft_model = PeftModel.from_pretrained(base, lora_adapter_path)
print("Merging LoRA weights into base model...")
merged = peft_model.merge_and_unload() # returns a standard nn.Module
print(f"Saving merged model to: {output_path}")
merged.save_pretrained(output_path)
AutoTokenizer.from_pretrained(base_model_name).save_pretrained(output_path)
before_mb = sum(p.numel() for p in peft_model.parameters()) * 2 / 1e6
after_mb = sum(p.numel() for p in merged.parameters()) * 2 / 1e6
print(f"Size before merge: ~{before_mb:.0f} MB (adapter only: ~{before_mb*0.002:.0f} MB)")
print(f"Size after merge: ~{after_mb:.0f} MB (full model, efficient for inference)")During experimentation, keep the LoRA adapter separate from the base model — you can swap adapters for different tasks without re-downloading the ~14GB base model each time. Before deploying to production, merge the adapter into the base model (merge_and_unload()) — the merged model runs faster because inference is one simple matrix multiply (W + ΔW) rather than two separate operations.
Real-World Spotlight: Fine-tuning Llama-3 8B for Customer Support
# ── Production QLoRA pipeline for customer support bot ──
# (Pseudo-code — exact API depends on your model/hardware)
from peft import LoraConfig, get_peft_model, TaskType
from transformers import (AutoModelForCausalLM, AutoTokenizer,
BitsAndBytesConfig, TrainingArguments)
import torch
# Configuration
MODEL_NAME = 'meta-llama/Llama-3.2-3B-Instruct' # 3B for demo; use 8B for production
LORA_RANK = 16
LORA_ALPHA = 32
TARGET_MODS = ['q_proj', 'v_proj', 'k_proj', 'o_proj'] # all attention projections
NUM_EPOCHS = 3
BATCH_SIZE = 4
# QLoRA: 4-bit quantized base + fp16 LoRA
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type='nf4',
bnb_4bit_compute_dtype=torch.bfloat16,
)
# Load quantized base model
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME, quantization_config=bnb_config)
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
# Configure LoRA
lora_config = LoraConfig(
r=LORA_RANK, lora_alpha=LORA_ALPHA,
target_modules=TARGET_MODS,
lora_dropout=0.05, bias='none',
task_type=TaskType.CAUSAL_LM
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 4,194,304 || all params: 3,212,820,480 || trainable%: 0.1305
print("System resources for this fine-tune:")
print(" GPU memory: ~6 GB (RTX 3080 sufficient)")
print(" Training time: ~45 min (1200 examples × 3 epochs)")
print(" Adapter size: ~32 MB (all LoRA weights)")
print("\nExpected results after fine-tuning:")
print(" Response style: matches company tone (formal, concise)")
print(" Product accuracy: knows product names, pricing, features")
print(" Routing accuracy: flags edge cases for human review")
print(" Perplexity (domain): ~18 (vs ~85 for base model on company docs)")
# ── LoRA vs RAG decision guide ──
print("\n── When to use LoRA vs RAG ──")
scenarios = [
("Change the model's tone/style", "✅ LoRA", "❌ RAG won't help"),
("Teach specialized vocabulary", "✅ LoRA", "⚠️ RAG can do this too"),
("Answer questions about recent events","❌ LoRA static", "✅ RAG (update index)"),
("Access proprietary factual database", "⚠️ LoRA needs retraining", "✅ RAG (update docs)"),
("Reduce hallucinations on domain facts","⚠️ Partial", "✅ RAG grounds answers"),
("Match brand writing style", "✅ LoRA", "❌ RAG"),
("Answer from a 100K doc knowledge base","❌ Too much for FT", "✅ RAG scales"),
("Combined (style + knowledge)", "✅ LoRA for style", "✅ RAG for facts"),
]
print(f"{'Scenario':<45} {'LoRA':>18} {'RAG':>25}")
print("─" * 90)
for scenario, lora_ans, rag_ans in scenarios:
print(f"{scenario:<45} {lora_ans:>18} {rag_ans:>25}")The best production systems often use both. LoRA fine-tuning gives the model the right personality, tone, and task-specific behavior (e.g., always format responses as bullet points, always suggest a support ticket if unresolved). RAG provides accurate, up-to-date factual grounding — preventing hallucinations about product specifications, pricing, or policies that change frequently. Think of LoRA as shaping who the model is, and RAG as shaping what the model knows.
Quick Check
✍️ Practice Exercises
- Implement a LoRA adapter from scratch (as in Section 3) and apply it to a
nn.Linearlayer in a pre-trained GPT-2. Verify that: (a) the LoRA output equals the original linear output at initialization (before any training), (b) after one gradient step on a loss, only A and B have updated values, (c) the merged weight (W + B@A*scaling) produces the same forward output as the LoRA layer. - Apply LoRA with ranks r ∈ [2, 4, 8, 16, 32] to GPT-2 and fine-tune each for 1 epoch on a small dataset. Plot test perplexity vs trainable parameter count. At what rank do you see diminishing returns? This is the rank vs quality tradeoff in practice.
- Use the
peftlibrary to fine-tune GPT-2 with LoRA on the SMS spam dataset from Lesson 54. Compare: (a) trainable parameter count, (b) training time, (c) final accuracy, vs the full BERT fine-tuning approach from Lesson 54. What are the tradeoffs? - Implement the LoRA weight merging operation manually: load a fine-tuned PEFT model, extract the lora_A and lora_B matrices for one layer, compute the merged weight matrix W_merged = W + B@A * scaling, and verify it equals the output of
peft_model.merge_and_unload()for that layer.
📚 Primary Sources for This Lesson
LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2022) — the original LoRA paper. Clear, concise, and full of useful ablations.
QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023) — introduced 4-bit NF4 quantization enabling 7B fine-tuning on consumer GPUs.
Hugging Face PEFT Documentation — comprehensive reference for all PEFT methods with worked examples.
Prefix-Tuning: Optimizing Continuous Prompts for Generation (Li & Liang, 2021) — the prefix tuning paper.