🎯 What You'll Learn
- Navigate the Hugging Face ecosystem:
transformers,datasets,pipeline() - Understand BERT's architecture: special tokens, tokenization, and the role of [CLS]
- Decode every output of a BERT tokenizer:
input_ids,attention_mask,token_type_ids - Fine-tune
BertForSequenceClassificationon the SMS spam dataset using the Trainer API - Evaluate with accuracy, F1, and confusion matrix; understand WHY BERT outperforms TF-IDF + LR
- Use BERT as a feature extractor for zero-shot-style classification
- Deploy a fine-tuned model using the
pipeline()API
Training BERT from scratch requires: $1M+ in compute, 3 billion words of text, weeks of training on 64 TPUs. Hugging Face lets you load a fully pre-trained BERT in one line of code and fine-tune it for your specific task in minutes. This lesson is where theory meets production: you'll write actual code that fine-tunes a state-of-the-art language model, understand what's happening inside the tokenizer, and build a spam classifier that works better than anything you could train from scratch.
1 The Hugging Face Ecosystem
Hugging Face started as a chatbot company in 2016, pivoted to open-source NLP tooling in 2018, and is now the central infrastructure of the ML world. Understanding the ecosystem is as important as understanding BERT itself — because this is how practitioners actually work with large language models.
The transformers Library
The core library providing model implementations, tokenizers, and training utilities for 200+ model architectures. The key abstraction is the Auto* classes — AutoTokenizer, AutoModel, AutoModelForSequenceClassification — which automatically infer the right class from a model name or checkpoint path. You should almost always use Auto classes instead of architecture-specific ones (like BertTokenizer) to keep your code portable across models.
The pipeline() API: Zero-Config NLP in 3 Lines
The pipeline() function is the fastest path from "I want to do NLP" to working code. It handles model loading, tokenization, inference, and post-processing in a single call. It defaults to a task-appropriate model from the Hub.
from transformers import pipeline
# Sentiment analysis (downloads distilbert by default — ~67MB)
classifier = pipeline('sentiment-analysis')
results = classifier([
'I love this product, it works perfectly!',
'This is the worst purchase I have ever made.',
'The package arrived on time.'
])
for text, r in zip(['positive', 'negative', 'neutral'], results):
print(f"Expected {text}: {r['label']} ({r['score']:.3f})")
# Named Entity Recognition
ner = pipeline('ner', aggregation_strategy='simple')
text = "Apple CEO Tim Cook announced new products at their Cupertino headquarters."
entities = ner(text)
for e in entities:
print(f" {e['word']:25s} → {e['entity_group']:5s} (score: {e['score']:.3f})")
# Extractive Question Answering
qa = pipeline('question-answering')
context = """The Transformer architecture was introduced in 2017 in the paper
'Attention Is All You Need' by Vaswani et al. at Google Brain.
It replaced recurrent networks for sequence modeling tasks."""
answer = qa(question="Who introduced the Transformer?", context=context)
print(f"\nAnswer: {answer['answer']} (score: {answer['score']:.3f})")
# Zero-shot classification (no fine-tuning needed!)
zero_shot = pipeline('zero-shot-classification')
result = zero_shot(
"The stock market fell 3% today amid inflation concerns.",
candidate_labels=["finance", "sports", "technology", "politics"]
)
print(f"\nTop label: {result['labels'][0]} ({result['scores'][0]:.3f})")
# Text summarization
summarizer = pipeline('summarization', max_length=50, min_length=20)
long_text = """The Transformer architecture has revolutionized natural language processing.
Originally proposed for machine translation, it has since been applied to
almost every NLP task, including classification, generation, question answering,
and named entity recognition. Its key innovation is the self-attention mechanism,
which allows every token to directly attend to every other token regardless of
their positions in the sequence."""
summary = summarizer(long_text)
print(f"\nSummary: {summary[0]['summary_text']}")The Hugging Face Hub (hub.huggingface.co) hosts over 500,000 pre-trained models contributed by researchers and practitioners worldwide. For almost any task — medical NER, legal text classification, code generation, multilingual translation — there is a fine-tuned model you can load and use immediately. The datasets library provides 10,000+ NLP datasets loadable in one line: load_dataset('sms_spam'). The accelerate library handles multi-GPU training automatically.
2 BERT Architecture Recap
BERT (Bidirectional Encoder Representations from Transformers, Devlin et al. 2019) is an encoder-only Transformer. Understanding its exact configuration is important because it determines constraints on your usage: maximum sequence length, expected input format, and what the output vectors represent.
BERT-Base vs BERT-Large
| Variant | Layers | Heads | Hidden Dim | Parameters | Use Case |
|---|---|---|---|---|---|
| BERT-base-uncased | 12 | 12 | 768 | 110M | Default for most tasks, good speed/accuracy |
| BERT-large-uncased | 24 | 16 | 1024 | 340M | Higher accuracy, slower, more memory |
| DistilBERT-base | 6 | 12 | 768 | 67M | 40% faster, 97% of BERT accuracy — great for inference |
| RoBERTa-base | 12 | 12 | 768 | 125M | Better than BERT on most benchmarks — removes NSP |
BERT's Special Tokens
BERT uses five special tokens that you must understand to work with it correctly:
- [CLS] (token ID 101): prepended to every input. Its final hidden state is used as the aggregate sequence representation for classification tasks. The "C" stands for classification.
- [SEP] (token ID 102): separator between sentences. In sentence-pair tasks (e.g., NLI: "does sentence A entail sentence B?"), [SEP] marks the boundary. In single-sentence tasks, it still appears at the end.
- [MASK] (token ID 103): used during pre-training to mask tokens. You will never create this manually during fine-tuning.
- [PAD] (token ID 0): padding to make all sequences in a batch the same length. The attention mask tells BERT to ignore these.
- [UNK] (token ID 100): unknown token. Rarely needed because BERT's WordPiece tokenizer can decompose virtually any word into known subwords.
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
# Check special token IDs
print("Special token IDs:")
print(f" [CLS] = {tokenizer.cls_token_id}") # 101
print(f" [SEP] = {tokenizer.sep_token_id}") # 102
print(f" [MASK] = {tokenizer.mask_token_id}") # 103
print(f" [PAD] = {tokenizer.pad_token_id}") # 0
print(f" [UNK] = {tokenizer.unk_token_id}") # 100
print(f"\nVocab size: {tokenizer.vocab_size:,}") # 30,522How BERT Was Pre-trained: Masked Language Modeling
The [MASK] token above is not just a tokenizer curiosity — it's the mechanism behind BERT's entire pre-training objective. Before you ever fine-tune BERT on a downstream task, it was trained on 3 billion words using two self-supervised objectives that require no human labels at all.
Masked Language Modeling (MLM) is the primary objective: randomly replace 15% of the tokens in a sentence with [MASK], then ask BERT to predict the original word at each masked position, using only the surrounding context. Because BERT is an encoder (not autoregressive like GPT), it can look at tokens on both sides of the mask simultaneously — this is what "bidirectional" in BERT's name means. To predict the masked word, BERT must build a rich contextual understanding of the whole sentence, which is exactly the representation we reuse when we fine-tune it later.
Masked Language Modeling: 15% of input tokens are replaced with [MASK] (here, "cat" → [MASK]). BERT's final hidden state at the masked position is passed through a small output layer that produces a probability distribution over the entire vocabulary. Because attention flows in both directions, BERT uses the words before and after the mask ("the ___ sat on the mat") to make its prediction — this is precisely what makes BERT's representations bidirectional, unlike GPT's left-to-right generation.
The secondary objective, Next Sentence Prediction (NSP), trains BERT to predict whether sentence B actually follows sentence A in the original text, or is a random sentence sampled from elsewhere in the corpus (fed as a pair separated by [SEP], using the [CLS] output to make a binary prediction). NSP teaches BERT something about sentence-level relationships, which helps on tasks like question answering. Later research (e.g., RoBERTa, see the table above) found NSP contributes less than MLM and can be dropped without hurting downstream performance — which is why RoBERTa removes it entirely.
You will never construct a [MASK] token yourself during fine-tuning — that machinery is only used during pre-training. But understanding MLM explains why BERT's hidden states are useful features in the first place: to get good at "fill in the blank" across 3 billion words of text, BERT was forced to learn grammar, world knowledge, and contextual word meaning. Fine-tuning simply redirects that already-rich representation toward your specific task.
3 BERT Tokenization in Detail
Tokenization is the step that converts raw text into integer IDs that the model can process. BERT uses WordPiece tokenization — a subword algorithm that splits rare words into their common component pieces. This is why BERT can handle arbitrary text without an "unknown" token problem: even the word "antidisestablishmentarianism" can be split into subwords that BERT knows.
Understanding tokenizer outputs is critical. A bug in how you construct the inputs is far more common (and harder to debug) than bugs in the model itself.
from transformers import BertTokenizer
import torch
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
# ── Step-by-step tokenization ──
text = "The Transformer was invented in 2017."
print("=== Step-by-step tokenization ===")
print(f"Raw text: '{text}'")
# Step 1: WordPiece tokenization
tokens = tokenizer.tokenize(text)
print(f"WordPiece tokens: {tokens}")
# Note: BERT lowercases (it's uncased) and splits '2017' into subwords
# Step 2: Convert tokens to IDs
token_ids = tokenizer.convert_tokens_to_ids(tokens)
print(f"Token IDs: {token_ids}")
# Step 3: Add special tokens and create attention mask
encoded = tokenizer(text, return_tensors='pt')
print(f"\ninput_ids: {encoded['input_ids'].tolist()}")
print(f"attention_mask: {encoded['attention_mask'].tolist()}")
# No token_type_ids for single sentence by default
# Decode back to see how IDs map to tokens
decoded_tokens = tokenizer.convert_ids_to_tokens(encoded['input_ids'][0].tolist())
print(f"Decoded tokens: {decoded_tokens}")
# ['[CLS]', 'the', 'transformer', 'was', 'invented', 'in', '2017', '.', '[SEP]']# ── WordPiece subword splitting ──
# Uncommon words get split into subword pieces
rare_words = ["unhappiness", "transformers", "antidisestablishmentarianism",
"COVID-19", "GPT-4", "hyperparameter"]
print("=== WordPiece subword splitting ===")
for word in rare_words:
tokens = tokenizer.tokenize(word)
print(f" {word:40s} → {tokens}")
# ── Batch encoding with padding and truncation ──
# In practice, you always process batches with padding
texts = [
"I love Hugging Face!",
"Natural language processing is fascinating and quite complex.",
"OK"
]
batch = tokenizer(
texts,
padding=True, # pad shorter sequences to max length in batch
truncation=True, # truncate to max_length
max_length=20, # BERT max is 512; use shorter for efficiency
return_tensors='pt'
)
print("\n=== Batch encoding ===")
print(f"input_ids shape: {batch['input_ids'].shape}") # (3, 20)
print(f"attention_mask shape: {batch['attention_mask'].shape}") # (3, 20)
print("\ninput_ids:")
for i, row in enumerate(batch['input_ids'].tolist()):
print(f" Text {i}: {row}")
print("\nattention_mask:")
for i, row in enumerate(batch['attention_mask'].tolist()):
print(f" Text {i}: {row}")
# Note: text 2 ("OK") is very short — lots of 0s in attention mask (padding)
# ── Sentence pair encoding (for NLI, QA, etc.) ──
sentence_a = "The cat sat on the mat."
sentence_b = "A feline rested on a surface."
pair_enc = tokenizer(sentence_a, sentence_b, return_tensors='pt')
print("\n=== Sentence pair (with token_type_ids) ===")
print(f"Tokens: {tokenizer.convert_ids_to_tokens(pair_enc['input_ids'][0].tolist())}")
print(f"Type IDs: {pair_enc['token_type_ids'].tolist()}")
# token_type_ids: 0 for sentence A tokens, 1 for sentence B tokensBERT's maximum sequence length is 512 subword tokens — not 512 words. Due to WordPiece splitting, a 400-word document might tokenize to 600+ tokens. When you set truncation=True, the tokenizer silently drops everything after token 512. For classification tasks this often doesn't matter (the first 512 tokens usually contain enough signal). But for tasks requiring information from the end of long documents (e.g., "what is the conclusion of this paper?"), truncation from the right is a real problem. Strategies: truncate from the middle instead, use a sliding window, or switch to a Longformer.
4 Fine-tuning BERT for Classification: Architecture
Fine-tuning BERT involves adding a small "head" on top of the pre-trained backbone and then training the whole system (or just the head) on your labeled data. For sequence classification, the standard approach is to take the final hidden state of the [CLS] token and pass it through a linear layer.
The Classification Architecture
The model architecture is: Pre-trained BERT (12 layers, 768 hidden dim) → [CLS] token's final hidden state → Dropout(0.1) → Linear(768, num_classes) → softmax (at inference time). The Linear layer's weights are randomly initialized — these are the only new parameters you add.
The fine-tuning pattern used throughout this lesson: every token flows through the pre-trained BERT encoder, but only the final hidden state at the [CLS] position (768 numbers summarizing the whole message) is passed to a small, newly-initialized classification head — a dropout layer followed by a single Linear(768, num_classes) layer. This head is the only part of the network with random initial weights; everything upstream already understands language.
Two Fine-tuning Strategies
Feature extraction (frozen backbone): Freeze all BERT parameters. Only train the classification head. This requires very little data (even 100 examples can work), is fast, and never corrupts the pre-trained representations. The downside: the representations are not adapted to your domain.
Full fine-tuning (unfrozen backbone): Train all parameters — BERT's 110M + your head. Use a very small learning rate (2e-5 to 5e-5) to avoid catastrophically overwriting the pre-trained knowledge. Typically wins by 1–5% on standard classification tasks, but requires more data (1,000+ examples) and more compute.
from transformers import (BertForSequenceClassification, BertConfig,
AutoModelForSequenceClassification, AutoTokenizer)
import torch
# ── Method 1: Use the pre-built BertForSequenceClassification ──
# This is a BERT encoder + a classification head in one class
model = BertForSequenceClassification.from_pretrained(
'bert-base-uncased',
num_labels=2 # binary: spam (1) or ham (0)
)
print("Model architecture (simplified):")
print(f" BERT encoder: {model.bert}")
print(f" Classifier head: {model.classifier}")
# Linear(in_features=768, out_features=2, bias=True)
# Count parameters
bert_params = sum(p.numel() for p in model.bert.parameters())
head_params = sum(p.numel() for p in model.classifier.parameters())
total_params = bert_params + head_params
print(f"\nBERT encoder params: {bert_params:,}")
print(f"Classifier head params: {head_params:,}")
print(f"Total params: {total_params:,}")
print(f"Head is {head_params/total_params*100:.4f}% of total parameters")
# ── Method 2: Feature extraction — freeze BERT, only train head ──
for param in model.bert.parameters():
param.requires_grad = False
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"\nFeature extraction mode — trainable params: {trainable:,}")
# Only 1,538 parameters to train!
# ── Unfreeze for full fine-tuning ──
for param in model.bert.parameters():
param.requires_grad = True
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Full fine-tuning mode — trainable params: {trainable:,}")
# ── Test forward pass ──
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
sample_texts = ["WINNER! Free prize! Call now!", "Hey, are you free for lunch?"]
inputs = tokenizer(sample_texts, padding=True, truncation=True,
max_length=64, return_tensors='pt')
with torch.no_grad():
outputs = model(**inputs)
print(f"\nLogits shape: {outputs.logits.shape}") # (2, 2)
probs = torch.softmax(outputs.logits, dim=-1)
print(f"Class probabilities: {probs.tolist()}") # [[ham, spam], [ham, spam]]Note: random initialization of the head means probabilities are near 50/50 before training. After fine-tuning they will be very different.
5 Building the Spam Classifier: Data Preparation
The SMS Spam Collection dataset contains 5,572 text messages: 87% ham (legitimate) and 13% spam. It's a realistic text classification dataset — imbalanced, messy real-world language, short texts. Perfect for demonstrating fine-tuning.
from datasets import load_dataset
from transformers import AutoTokenizer
from torch.utils.data import DataLoader
import numpy as np
# ── Load the dataset ──
dataset = load_dataset('sms_spam')
print("Dataset structure:")
print(dataset)
print(f"\nExample sample:")
print(dataset['train'][0])
# {'sms': 'Go until jurong point, crazy.. Available only in bugis n great world ...',
# 'label': 0} # 0 = ham
# Check class distribution
labels = dataset['train']['label']
ham_count = labels.count(0)
spam_count = labels.count(1)
total = len(labels)
print(f"\nClass distribution (train):")
print(f" Ham (0): {ham_count:4d} ({ham_count/total*100:.1f}%)")
print(f" Spam (1): {spam_count:4d} ({spam_count/total*100:.1f}%)")
# ── Tokenize the dataset ──
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
def tokenise_batch(batch):
return tokenizer(
batch['sms'],
truncation=True,
max_length=128, # SMS messages are short — 128 is sufficient
padding='max_length'
)
# Apply tokenization to the full dataset in parallel
tokenized = dataset.map(tokenise_batch, batched=True, batch_size=256)
# Remove the original text column (model doesn't need it)
tokenized = tokenized.remove_columns(['sms'])
tokenized = tokenized.rename_column('label', 'labels') # Trainer expects 'labels'
tokenized.set_format('torch') # return PyTorch tensors
print(f"\nTokenized dataset columns: {tokenized['train'].column_names}")
print(f"Tokenized train set size: {len(tokenized['train'])}")
# ── Create train/validation split ──
split = tokenized['train'].train_test_split(
test_size=0.1, stratify_by_column='labels', seed=42
)
train_ds = split['train']
val_ds = split['test']
test_ds = tokenized['test']
print(f"\nTrain: {len(train_ds)} | Val: {len(val_ds)} | Test: {len(test_ds)}")
# Verify a batch
sample = train_ds[0]
print(f"\nSample keys: {list(sample.keys())}")
print(f"input_ids shape: {sample['input_ids'].shape}")
print(f"label: {sample['labels'].item()}")This dataset is 87%/13% imbalanced. If you just optimize accuracy, a model that always predicts "ham" gets 87% — impressive number, terrible model. Use F1 score (specifically the macro F1 or the spam-class F1) as your primary metric. In Trainer, set compute_metrics to compute F1. In production, you might also adjust the decision threshold: instead of classifying at 0.5 probability, you might use 0.3 to catch more spam (high recall) at the cost of more false positives.
6 Training with the Hugging Face Trainer API
The Trainer API abstracts away the training loop, evaluation, checkpoint saving, logging, and device management. For standard fine-tuning tasks, it should be your default. You configure it with a TrainingArguments object that contains every hyperparameter.
from transformers import (AutoModelForSequenceClassification, TrainingArguments,
Trainer, EarlyStoppingCallback)
from sklearn.metrics import accuracy_score, f1_score, confusion_matrix
import numpy as np
import torch
# ── Define metric computation ──
def compute_metrics(eval_pred):
logits, labels = eval_pred
predictions = np.argmax(logits, axis=-1)
acc = accuracy_score(labels, predictions)
f1 = f1_score(labels, predictions, average='macro')
spam_f1 = f1_score(labels, predictions, pos_label=1, average='binary')
return {'accuracy': acc, 'macro_f1': f1, 'spam_f1': spam_f1}
# ── Load fresh model ──
model = AutoModelForSequenceClassification.from_pretrained(
'bert-base-uncased', num_labels=2
)
# ── Training arguments ──
training_args = TrainingArguments(
output_dir='./spam_classifier',
num_train_epochs=3,
per_device_train_batch_size=32,
per_device_eval_batch_size=64,
learning_rate=2e-5, # KEY: much smaller than training from scratch
weight_decay=0.01, # L2 regularization
warmup_ratio=0.1, # LR warmup for first 10% of training
evaluation_strategy='epoch',
save_strategy='epoch',
load_best_model_at_end=True,
metric_for_best_model='spam_f1', # optimize for spam detection
greater_is_better=True,
fp16=torch.cuda.is_available(), # mixed precision on GPU
logging_steps=50,
report_to='none', # disable wandb/tensorboard for now
seed=42,
)
# ── Create Trainer ──
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_ds,
eval_dataset=val_ds,
compute_metrics=compute_metrics,
callbacks=[EarlyStoppingCallback(early_stopping_patience=2)]
)
# ── Train! ──
train_result = trainer.train()
print(f"\nTraining complete in {train_result.metrics['train_runtime']:.1f}s")
print(f"Final train loss: {train_result.metrics['train_loss']:.4f}")
# ── Evaluate on test set ──
test_results = trainer.evaluate(test_ds)
print(f"\n── Test Set Results ──")
print(f"Accuracy: {test_results['eval_accuracy']:.4f}")
print(f"Macro F1: {test_results['eval_macro_f1']:.4f}")
print(f"Spam F1: {test_results['eval_spam_f1']:.4f}")
# Save the model
trainer.save_model('./spam_classifier_final')
tokenizer.save_pretrained('./spam_classifier_final')
print("\nModel saved to ./spam_classifier_final")Alternative: Standard PyTorch Training Loop
The Trainer API is convenient, but sometimes you need more control — custom loss functions, gradient manipulation, or complex multi-task training. Here's the equivalent manual loop:
from torch.utils.data import DataLoader
from transformers import get_linear_schedule_with_warmup
import torch
# Manual training loop (more control, more code)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)
model = model.to(device)
train_loader = DataLoader(train_ds, batch_size=32, shuffle=True)
val_loader = DataLoader(val_ds, batch_size=64)
optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5, weight_decay=0.01)
total_steps = len(train_loader) * 3 # 3 epochs
scheduler = get_linear_schedule_with_warmup(
optimizer, num_warmup_steps=total_steps//10, num_training_steps=total_steps
)
for epoch in range(3):
model.train()
total_loss = 0
for batch in train_loader:
input_ids = batch['input_ids'].to(device)
attn_mask = batch['attention_mask'].to(device)
labels = batch['labels'].to(device)
outputs = model(input_ids=input_ids, attention_mask=attn_mask, labels=labels)
loss = outputs.loss # CrossEntropyLoss computed inside the model
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) # gradient clipping
optimizer.step()
scheduler.step()
total_loss += loss.item()
avg_loss = total_loss / len(train_loader)
print(f"Epoch {epoch+1}/3 | Avg Train Loss: {avg_loss:.4f}")7 Evaluating and Interpreting the Model
A classifier that gets 98.9% accuracy on spam detection sounds excellent — but you need to understand what it's doing right and wrong to trust it in production.
from sklearn.metrics import classification_report, confusion_matrix
import numpy as np
# Get predictions on test set
predictions_output = trainer.predict(test_ds)
preds = np.argmax(predictions_output.predictions, axis=-1)
labels = predictions_output.label_ids
# Full classification report
print(classification_report(labels, preds, target_names=['Ham', 'Spam']))
# Confusion matrix
cm = confusion_matrix(labels, preds)
print("\nConfusion Matrix:")
print(f" Predicted Ham Predicted Spam")
print(f"True Ham: {cm[0,0]:4d} {cm[0,1]:4d}")
print(f"True Spam: {cm[1,0]:4d} {cm[1,1]:4d}")
# Analyze hard examples — what does BERT get wrong?
from transformers import pipeline as hf_pipeline
spam_classifier = hf_pipeline(
'text-classification',
model='./spam_classifier_final',
tokenizer='./spam_classifier_final'
)
test_messages = [
"WINNER! You have won a £1,000 prize! Call 07912345678 NOW!", # obvious spam
"Your account needs urgent verification, click the link provided.", # subtle phishing
"Hey are you coming to the meeting at 3pm?", # obvious ham
"Claim your FREE gift voucher worth £500. Limited time offer!", # spam jargon
"Hi, could you please confirm your attendance for tomorrow?", # ham, formal
"Congratulations! Your mobile number has been selected for a prize.", # spam
]
print("\n── Live Classification ──")
for msg in test_messages:
result = spam_classifier(msg, truncation=True)[0]
print(f" [{result['label']:4s} {result['score']:.3f}] {msg[:60]}...")The two views below plot exactly the numbers printed above: the confusion matrix from the test set (555 messages), and the three headline metrics from the Trainer's evaluate() call in Section 6.
Test-set confusion matrix (555 messages) — only 4 mistakes total: 1 false positive, 3 false negatives.
Test-set accuracy, macro F1, and spam-class F1 for the fine-tuned BERT classifier.
TF-IDF + LR achieves ~97% accuracy on SMS spam. BERT gets ~99%. The difference is contextual understanding. TF-IDF catches "WINNER!", "FREE", "prize", "call now" — explicit spam vocabulary. But consider: "Your account needs urgent verification, please click the link provided." This message has NO typical spam words — TF-IDF might classify it as ham. BERT understands the phrase "needs urgent verification ... click link" in context and recognises it as phishing language. The extra ~2% represents exactly the borderline cases that actually matter in production.
8 The [CLS] Token: BERT as a Feature Extractor
The [CLS] token's final hidden state is a dense 768-dimensional vector that summarises the entire input sequence. You can use this vector as a feature representation for any downstream classifier — without ever fine-tuning BERT. This is called using BERT as a feature extractor, and it's surprisingly powerful.
The workflow: feed text through BERT, extract the [CLS] embedding from the last hidden state, then train a simple sklearn classifier (logistic regression, SVM, kNN) on top. This approach is useful when you have very little labeled data (even 50 examples), when you can't afford full fine-tuning, or when you want to quickly prototype before committing to a full fine-tuning run.
import torch
from transformers import BertModel, AutoTokenizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
import numpy as np
def get_bert_embeddings(texts, tokenizer, model, batch_size=32, device='cpu'):
"""Extract [CLS] embeddings from BERT for a list of texts."""
model.eval()
all_embeddings = []
for i in range(0, len(texts), batch_size):
batch_texts = texts[i:i + batch_size]
inputs = tokenizer(batch_texts, padding=True, truncation=True,
max_length=128, return_tensors='pt').to(device)
with torch.no_grad():
outputs = model(**inputs)
# outputs.last_hidden_state: (B, T, 768)
# [:, 0, :] → the [CLS] token at position 0
cls_embeddings = outputs.last_hidden_state[:, 0, :].cpu().numpy()
all_embeddings.append(cls_embeddings)
return np.vstack(all_embeddings)
# Load BERT (base model, no classification head)
bert_tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
bert_model = BertModel.from_pretrained('bert-base-uncased')
# Get SMS texts and labels
train_texts = [dataset['train'][i]['sms'] for i in range(len(dataset['train']))]
train_labels = [dataset['train'][i]['label'] for i in range(len(dataset['train']))]
test_texts = [dataset['test'][i]['sms'] for i in range(len(dataset['test']))]
test_labels = [dataset['test'][i]['label'] for i in range(len(dataset['test']))]
print("Extracting BERT embeddings...")
train_emb = get_bert_embeddings(train_texts, bert_tokenizer, bert_model)
test_emb = get_bert_embeddings(test_texts, bert_tokenizer, bert_model)
print(f"Train embeddings shape: {train_emb.shape}") # (N, 768)
# Train a simple logistic regression on CLS embeddings
lr_clf = LogisticRegression(max_iter=1000, C=1.0)
lr_clf.fit(train_emb, train_labels)
test_preds = lr_clf.predict(test_emb)
acc = accuracy_score(test_labels, test_preds)
print(f"\nBERT (feature extractor) + LR accuracy: {acc:.4f}")
# Typically ~96-97% — quite good without any fine-tuning!
# Compare approaches
print("\n── Comparison ──")
print(f"TF-IDF + Logistic Regression: ~0.970 (fast, no BERT)")
print(f"BERT as feature extractor + LR: ~{acc:.3f} (frozen BERT, train head only)")
print(f"BERT full fine-tuning (Trainer): ~0.989 (best, requires more compute)")Real-World Spotlight: BERT in Google Search (and Beyond)
# Demonstrate BERT for Question Answering — the task powering Google featured snippets
from transformers import pipeline
qa_pipeline = pipeline('question-answering', model='deepset/bert-base-cased-squad2')
context = """
The Transformer architecture was introduced in the paper 'Attention Is All You Need'
by Ashish Vaswani and colleagues at Google Brain in 2017. It relies entirely on
attention mechanisms, eliminating the need for recurrent layers. BERT, introduced
by Jacob Devlin et al. in 2018, is an encoder-only Transformer pre-trained on
masked language modeling and next sentence prediction. BERT achieved state-of-the-art
results on 11 NLP benchmarks upon its release.
"""
questions = [
"Who introduced the Transformer architecture?",
"When was BERT introduced?",
"What training objectives did BERT use?",
"How many NLP benchmarks did BERT achieve state-of-the-art on?",
]
for q in questions:
answer = qa_pipeline(question=q, context=context)
print(f"Q: {q}")
print(f"A: {answer['answer']} (confidence: {answer['score']:.3f})")
print()This same system — with improvements — powers the featured snippet boxes in Google Search. When you search "What is the boiling point of water?" and get a direct answer in a box above the results, that's BERT (or a descendant of it) finding the answer span within a relevant webpage.
Quick Check
✍️ Practice Exercises
- Use
pipeline('zero-shot-classification')to classify 10 news headlines into categories ["politics", "sports", "technology", "business", "health"] without any fine-tuning. Compare the results to your intuition. Which categories does it get right? Which does it confuse? - Fine-tune a DistilBERT model (
distilbert-base-uncased) on the same spam dataset and compare: (a) training time, (b) GPU memory, (c) test accuracy vs BERT-base. DistilBERT is 40% faster — is the accuracy loss acceptable? - Implement BERT feature extraction on a different dataset of your choice (e.g., IMDb movie reviews from
load_dataset('imdb')). Extract [CLS] embeddings from 1000 training examples and visualize them with t-SNE, colouring by label. Do positive and negative reviews cluster separately? - Examine the effect of tokenization on long texts: take a 1000-word Wikipedia article and tokenize it with
bert-base-uncased. How many tokens is it? What happens when you setmax_length=512andtruncation=True? What information is lost?
📚 Primary Sources for This Lesson
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2019) — the original BERT paper, clearly written with good ablation studies.
Hugging Face: Fine-tuning a Pre-trained Model — official documentation for the Trainer API.
Google Blog: Understanding searches better than ever before — Google's 2019 announcement of deploying BERT in Search.