🎯 What You'll Learn
- Read and interpret a confusion matrix — the foundation of all classification evaluation
- Understand TP, TN, FP, FN and why accuracy is dangerously misleading for imbalanced classes
- Compute and interpret Precision, Recall (Sensitivity), F1 Score, and F-Beta Score
- Distinguish macro, micro, and weighted averaging for multi-class metrics
- Apply a business-driven framework to choose the right metric for any classification problem
1 The Confusion Matrix
Before any single number metric, look at the confusion matrix. For binary classification, it's a 2×2 table showing the full breakdown of predictions vs actual labels:
| Predicted: Positive | Predicted: Negative | |
|---|---|---|
| Actual: Positive | TP — True Positive | FN — False Negative (Type II) |
| Actual: Negative | FP — False Positive (Type I) | TN — True Negative |
- True Positive (TP): Model predicts Positive. Actual is Positive. Correct.
- True Negative (TN): Model predicts Negative. Actual is Negative. Correct.
- False Positive (FP): Model predicts Positive. Actual is Negative. Type I error — a false alarm.
- False Negative (FN): Model predicts Negative. Actual is Positive. Type II error — a missed detection.
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np
import matplotlib.pyplot as plt
cancer = load_breast_cancer()
X, y = cancer.data, cancer.target # 0=malignant, 1=benign
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
pipe = Pipeline([
('scaler', StandardScaler()),
('lr', LogisticRegression(max_iter=1000, random_state=42))
])
pipe.fit(X_tr, y_tr)
y_pred = pipe.predict(X_te)
# Confusion matrix
cm = confusion_matrix(y_te, y_pred)
print("Confusion Matrix:")
print(cm)
print(f"\n TN={cm[0,0]} FP={cm[0,1]}")
print(f" FN={cm[1,0]} TP={cm[1,1]}")
# Visual display
fig, ax = plt.subplots(figsize=(5, 4))
ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=cancer.target_names).plot(ax=ax)
plt.title("Confusion Matrix — Breast Cancer Classifier")
plt.tight_layout()
plt.show()
# Extract individual counts
tn, fp, fn, tp = cm.ravel()
print(f"\nTrue Negatives (malignant correctly identified): {tn}")
print(f"False Positives (malignant incorrectly benign): {fp}")
print(f"False Negatives (benign incorrectly malignant): {fn}")
print(f"True Positives (benign correctly identified): {tp}")2 Accuracy and Its Limitations
Accuracy = (TP + TN) / (TP + TN + FP + FN). The fraction of all predictions that were correct. It's the most intuitive metric and perfectly fine when classes are balanced. But it becomes deeply misleading for imbalanced datasets.
Consider a dataset for fraud detection: 99% of transactions are legitimate, 1% are fraud. A model that predicts "not fraud" for every single transaction achieves 99% accuracy — a number that sounds great but represents a completely useless model that catches zero frauds.
If your dataset has a dominant class (say, 95%+ of one class), accuracy is nearly meaningless. A trivial model that always predicts the majority class will have high accuracy. Always examine the full classification report and confusion matrix. The first question when evaluating any classification model should be: "Are my classes balanced?"
import numpy as np
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
# Simulate a highly imbalanced dataset: 99% negative, 1% positive (e.g., rare disease)
np.random.seed(42)
n_samples = 10000
y_true = np.zeros(n_samples, dtype=int)
y_true[:100] = 1 # 100 positives out of 10,000
# Strategy 1: Predict everything as negative
y_pred_lazy = np.zeros(n_samples, dtype=int)
# Strategy 2: A decent classifier (but imperfect)
y_pred_model = y_true.copy()
y_pred_model[:40] = 0 # misses 40% of positives (40 FN)
y_pred_model[200:220] = 1 # generates 20 false alarms (20 FP)
print("=== Lazy 'always negative' model ===")
print(f"Accuracy: {accuracy_score(y_true, y_pred_lazy):.4f}") # 99% !!
print(f"True positives caught: {((y_pred_lazy == 1) & (y_true == 1)).sum()}")
print(f"Confusion matrix:\n{confusion_matrix(y_true, y_pred_lazy)}")
print("\n=== Real model ===")
print(f"Accuracy: {accuracy_score(y_true, y_pred_model):.4f}")
print(f"True positives caught: {((y_pred_model == 1) & (y_true == 1)).sum()}")
print(f"Confusion matrix:\n{confusion_matrix(y_true, y_pred_model)}")
print("\nClassification Report (Real Model):")
print(classification_report(y_true, y_pred_model, target_names=['healthy', 'disease']))The two confusion matrices below come straight from the code above (10,000 transactions, 100 actual positives). Toggle between the lazy model and the real model to see how identical-looking "high accuracy" hides very different behavior:
Lazy model: predicts "negative" for all 10,000 transactions. Accuracy = 99.00% — but it catches zero positives.
Now compare all four metrics side by side for both models. Accuracy barely moves (99.0% → 99.4%) while precision, recall, and F1 reveal that the lazy model is completely useless and the real model is actually working:
Accuracy looks similar for both models (99.0% vs 99.4%) — but Precision, Recall, and F1 are 0 for the lazy model and meaningfully positive for the real model. This is the "accuracy paradox" made visible.
3 Precision
Precision = TP / (TP + FP). Of all the samples the model predicted as Positive, what fraction were actually Positive?
Precision answers: "When the model raises an alarm, how often is it correct?" High precision means few false alarms. Low precision means the model cries wolf too often.
When precision matters most: Situations where false positives are costly.
- Email spam filter: A false positive means a legitimate email gets deleted. The user misses an important message. High precision is critical — you'd rather let some spam through than delete real emails.
- Content moderation: Wrongly removing a legitimate post damages user trust. High precision preferred.
- Drug discovery: Sending a non-effective compound to expensive clinical trials wastes millions of dollars. Want high precision on "promising compound" predictions.
from sklearn.metrics import precision_score, confusion_matrix
import numpy as np
# Manual calculation
y_true = np.array([1, 1, 0, 1, 0, 0, 1, 0, 1, 0])
y_pred = np.array([1, 1, 1, 1, 0, 1, 0, 0, 1, 0])
tp = ((y_pred == 1) & (y_true == 1)).sum()
fp = ((y_pred == 1) & (y_true == 0)).sum()
fn = ((y_pred == 0) & (y_true == 1)).sum()
tn = ((y_pred == 0) & (y_true == 0)).sum()
precision_manual = tp / (tp + fp)
print(f"TP={tp}, FP={fp}, FN={fn}, TN={tn}")
print(f"Precision (manual): {precision_manual:.4f}")
print(f"Precision (sklearn): {precision_score(y_true, y_pred):.4f}")
# Visualize precision in context
print("\nInterpretation:")
print(f"Model predicted {tp + fp} samples as positive")
print(f"Of these, {tp} were actually positive")
print(f"Precision = {tp}/{tp+fp} = {precision_manual:.2%}")4 Recall (Sensitivity)
Recall = TP / (TP + FN). Of all the actual Positive samples, what fraction did the model correctly identify?
Recall answers: "Of all the real positives, how many did we catch?" High recall means few missed detections. Low recall means many positives were missed.
When recall matters most: Situations where false negatives are costly.
- Cancer screening: A false negative means a cancer goes undetected. The patient doesn't get treatment. This can be fatal. High recall is critical — you'd rather send some healthy patients for follow-up tests than miss a cancer.
- Fraud detection: A missed fraud means money is lost and the criminal isn't caught. High recall preferred (though a balance with precision is needed).
- Airport security: Missing a dangerous item (FN) is catastrophic. False alarms (FP) are merely inconvenient. Extreme recall priority.
from sklearn.metrics import recall_score
import numpy as np
y_true = np.array([1, 1, 0, 1, 0, 0, 1, 0, 1, 0])
y_pred = np.array([1, 1, 1, 1, 0, 1, 0, 0, 1, 0])
tp = ((y_pred == 1) & (y_true == 1)).sum()
fn = ((y_pred == 0) & (y_true == 1)).sum()
recall_manual = tp / (tp + fn)
print(f"TP={tp}, FN={fn}")
print(f"Recall (manual): {recall_manual:.4f}")
print(f"Recall (sklearn): {recall_score(y_true, y_pred):.4f}")
print(f"\nInterpretation:")
print(f"There are {tp + fn} actual positive samples")
print(f"Model found {tp} of them")
print(f"Missed {fn} (false negatives)")
print(f"Recall = {tp}/{tp+fn} = {recall_manual:.2%}")
# The precision/recall tension: demonstrate with threshold adjustment
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import precision_score, recall_score
cancer = load_breast_cancer()
X_tr, X_te, y_tr, y_te = train_test_split(
cancer.data, cancer.target, test_size=0.2, random_state=42, stratify=cancer.target)
pipe = Pipeline([('s', StandardScaler()), ('lr', LogisticRegression(max_iter=1000))])
pipe.fit(X_tr, y_tr)
proba = pipe.predict_proba(X_te)[:, 1]
print(f"\n{'Threshold':>10} {'Precision':>10} {'Recall':>8}")
for t in [0.2, 0.3, 0.5, 0.7, 0.85]:
y_pred_t = (proba >= t).astype(int)
p = precision_score(y_te, y_pred_t, zero_division=0)
r = recall_score(y_te, y_pred_t, zero_division=0)
print(f"{t:>10.2f} {p:>10.4f} {r:>8.4f}")This is the precision-recall tradeoff in action: as the decision threshold rises, the model becomes pickier about calling something "positive" — precision climbs, but recall falls because more true positives get missed. Drag the slider to pick a threshold on the actual breast-cancer model above and watch both curves and the confusion counts move:
At threshold 0.50: Precision = 0.986, Recall = 0.986 — both high, close to where the curves cross.
5 F1 Score
Precision and recall are in tension: increasing one often decreases the other. The F1 score combines them into a single number that balances both:
F1 = 2 × (Precision × Recall) / (Precision + Recall)
F1 is the harmonic mean of precision and recall. Unlike the arithmetic mean, the harmonic mean is dominated by the lower of the two values. This means F1 is only high when both precision and recall are high. A model with precision=1.0 and recall=0.1 gets F1 = 2×(1.0×0.1)/(1.0+0.1) = 0.18 — rightfully penalized for missing almost all positives.
from sklearn.metrics import f1_score, precision_score, recall_score
import numpy as np
def f1_manual(precision, recall):
if precision + recall == 0:
return 0.0
return 2 * (precision * recall) / (precision + recall)
# Demonstrate harmonic mean effect
scenarios = [
("Balanced", 0.80, 0.80),
("High P, Low R", 1.00, 0.10),
("Low P, High R", 0.10, 1.00),
("Both moderate", 0.70, 0.60),
("Perfect classifier",1.00, 1.00),
]
print(f"{'Scenario':25} {'Precision':>10} {'Recall':>8} {'F1':>8} {'Arith. Mean':>12}")
for name, p, r in scenarios:
f1 = f1_manual(p, r)
avg = (p + r) / 2
print(f"{name:25} {p:>10.2f} {r:>8.2f} {f1:>8.4f} {avg:>12.4f}")
# sklearn f1_score with actual predictions
y_true = np.array([1, 1, 0, 1, 0, 0, 1, 0, 1, 0])
y_pred = np.array([1, 1, 1, 1, 0, 1, 0, 0, 1, 0])
p = precision_score(y_true, y_pred)
r = recall_score(y_true, y_pred)
f1_sk = f1_score(y_true, y_pred)
print(f"\nPrecision={p:.4f} Recall={r:.4f} F1={f1_sk:.4f}")
print(f"Manual F1={f1_manual(p, r):.4f} ✓")Use F1 (or precision/recall separately) whenever: (1) your classes are imbalanced, (2) different types of errors have different costs, or (3) you need to communicate clearly about false positives and false negatives. Use accuracy only when classes are balanced AND false positives and false negatives have equal cost — which is rare in production ML.
6 F-Beta Score
F1 weights precision and recall equally. But in many applications, one matters more. The F-Beta score generalises F1 by adding a β parameter that controls the relative weight:
F_β = (1 + β²) × (Precision × Recall) / (β² × Precision + Recall)
- β = 1: F1 — equal weight to precision and recall
- β = 2: F2 — weights recall 2× more than precision (use when missing positives is more costly)
- β = 0.5: F0.5 — weights precision 2× more than recall (use when false positives are more costly)
from sklearn.metrics import fbeta_score, f1_score
import numpy as np
y_true = np.array([1, 1, 1, 1, 1, 0, 0, 0, 0, 0])
y_pred = np.array([1, 1, 0, 0, 0, 0, 0, 0, 0, 1])
# This model has: TP=2, FN=3, FP=1, TN=4
# Precision = 2/(2+1) = 0.667, Recall = 2/(2+3) = 0.4
print(f"F1 (β=1.0): {f1_score(y_true, y_pred):.4f}")
print(f"F0.5 (β=0.5): {fbeta_score(y_true, y_pred, beta=0.5):.4f}") # precision-heavy
print(f"F2 (β=2.0): {fbeta_score(y_true, y_pred, beta=2.0):.4f}") # recall-heavy
# Practical use case: cancer screening
# FN (missing cancer) >> FP (unnecessary follow-up) → use F2
print("\n--- Cancer Screening Context ---")
print("Using F2 (recall matters 4x more than precision):")
print(f" F2 score: {fbeta_score(y_true, y_pred, beta=2.0):.4f}")
print("Using F0.5 (precision matters 4x more, e.g., spam filter):")
print(f" F0.5 score: {fbeta_score(y_true, y_pred, beta=0.5):.4f}")7 Multi-class Metrics: Macro, Micro, Weighted
For multi-class problems (K > 2), we compute per-class precision, recall, and F1, then average them. sklearn offers three averaging strategies:
- Macro-average: Compute the metric for each class independently, then take the unweighted mean. Treats all classes equally regardless of size. Useful when you care about performance on minority classes.
- Weighted-average: Compute the metric for each class, then take the weighted mean, where weights are the number of samples in each class (support). Accounts for class imbalance — a metric close to accuracy on the most common class.
- Micro-average: Aggregate TP, FP, FN globally across all classes, then compute the metric. For precision and recall, micro-average gives equal weight to each sample regardless of class. For balanced data, micro ≈ macro.
from sklearn.metrics import classification_report, precision_score, recall_score, f1_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np
digits = load_digits()
X, y = digits.data, digits.target
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y)
pipe = Pipeline([
('scaler', StandardScaler()),
('lr', LogisticRegression(max_iter=5000, multi_class='multinomial'))
])
pipe.fit(X_tr, y_tr)
y_pred = pipe.predict(X_te)
# Full classification report
print("Classification Report (10-class digit recognition):")
print(classification_report(y_te, y_pred, digits=4))
# Manual comparison of averaging methods
for avg in ['macro', 'weighted', 'micro']:
p = precision_score(y_te, y_pred, average=avg)
r = recall_score(y_te, y_pred, average=avg)
f = f1_score(y_te, y_pred, average=avg)
print(f"{avg:10} Precision={p:.4f} Recall={r:.4f} F1={f:.4f}")
# Per-class breakdown (without averaging)
per_class_f1 = f1_score(y_te, y_pred, average=None)
print("\nPer-class F1 scores:")
for digit, f1 in enumerate(per_class_f1):
bar = '█' * int(f1 * 20)
print(f" Digit {digit}: {f1:.4f} {bar}")8 Choosing the Right Metric
The right metric is a business decision, not a technical one. Ask: "What does a false positive cost? What does a false negative cost?" This comparison determines whether to prioritize precision, recall, F1, or a custom F-Beta.
| Application | FP Cost | FN Cost | Recommended Metric |
|---|---|---|---|
| Cancer screening | Unnecessary follow-up (minor) | Missed cancer (potentially fatal) | Recall, F2 |
| Email spam | Legitimate email deleted | Spam in inbox (minor) | Precision, F0.5 |
| Fraud detection | Manual review cost ($50) | Fraud loss ($500+) | F1 or custom threshold |
| Recommendation system | Bad recommendation (mild annoyance) | Missed good recommendation | Precision@K, NDCG |
| Quality control (defects) | Good part discarded (waste) | Defective part shipped (recall) | Depends on cost ratio |
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import precision_score, recall_score, f1_score, fbeta_score
import numpy as np
cancer = load_breast_cancer()
# Reframe: class 0 = malignant (positive), class 1 = benign
X, y = cancer.data, 1 - cancer.target # flip so malignant=1
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
pipe = Pipeline([
('s', StandardScaler()),
('lr', LogisticRegression(max_iter=1000))
])
pipe.fit(X_tr, y_tr)
proba = pipe.predict_proba(X_te)[:, 1]
print("Finding the threshold that maximises recall (fewest missed cancers):")
print(f"\n{'Threshold':>10} {'Precision':>10} {'Recall':>8} {'F1':>8} {'F2':>8} {'FN (missed)':>12}")
for t in np.arange(0.1, 0.95, 0.1):
y_pred_t = (proba >= t).astype(int)
p = precision_score(y_te, y_pred_t, zero_division=0)
r = recall_score(y_te, y_pred_t, zero_division=0)
f1 = f1_score(y_te, y_pred_t, zero_division=0)
f2 = fbeta_score(y_te, y_pred_t, beta=2, zero_division=0)
fn = ((y_pred_t == 0) & (y_te == 1)).sum()
print(f"{t:>10.2f} {p:>10.4f} {r:>8.4f} {f1:>8.4f} {f2:>8.4f} {fn:>12}")Real-World Spotlight: Cancer Diagnosis Model
Imagine deploying a cancer detection model that achieves 97% accuracy. The medical team is impressed. But looking at the confusion matrix reveals the truth: the dataset is 97% healthy patients, 3% cancer. The model learned to predict "healthy" for everyone — 97% accuracy, zero cancer detected.
Switching the optimization target to F2 score (which values recall 4× more than precision) forces the model to prioritize finding cancer cases even at the cost of more false alarms. A model with 90% recall and 60% precision (F2 = 0.83) is vastly more useful clinically than a 97% accurate model with 50% recall.
The lesson: always align your training objective and evaluation metric with the real business/clinical cost of each type of error.
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import (accuracy_score, precision_score, recall_score,
f1_score, fbeta_score, confusion_matrix)
import numpy as np
# Imbalanced dataset simulating cancer screening (3% positive)
X, y = make_classification(n_samples=5000, weights=[0.97, 0.03],
n_features=20, n_informative=10,
random_state=42, flip_y=0.02)
print(f"Class distribution: {np.bincount(y)} (positive rate: {y.mean():.2%})")
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.3, stratify=y, random_state=42)
# Model A: default settings, optimized for accuracy
pipe_a = Pipeline([('s', StandardScaler()),
('lr', LogisticRegression(max_iter=1000))])
pipe_a.fit(X_tr, y_tr)
y_pred_a = pipe_a.predict(X_te)
# Model B: class_weight='balanced' — upweights minority class
pipe_b = Pipeline([('s', StandardScaler()),
('lr', LogisticRegression(max_iter=1000, class_weight='balanced'))])
pipe_b.fit(X_tr, y_tr)
y_pred_b = pipe_b.predict(X_te)
print("\n=== Model A (default) ===")
cm_a = confusion_matrix(y_te, y_pred_a)
print(f"Accuracy: {accuracy_score(y_te, y_pred_a):.4f}")
print(f"Recall : {recall_score(y_te, y_pred_a):.4f} (caught {cm_a[1,1]} of {cm_a[1,:].sum()} cancers)")
print(f"F2 score: {fbeta_score(y_te, y_pred_a, beta=2):.4f}")
print("\n=== Model B (class_weight='balanced') ===")
cm_b = confusion_matrix(y_te, y_pred_b)
print(f"Accuracy: {accuracy_score(y_te, y_pred_b):.4f}")
print(f"Recall : {recall_score(y_te, y_pred_b):.4f} (caught {cm_b[1,1]} of {cm_b[1,:].sum()} cancers)")
print(f"F2 score: {fbeta_score(y_te, y_pred_b, beta=2):.4f}")Quick Check
✍️ Practice Exercises
- Load the
load_breast_cancer()dataset. Train a logistic regression model. Print the confusion matrix andclassification_report. Identify the FP and FN counts and explain in plain English what each means clinically. - Create a severely imbalanced dataset with
make_classification(weights=[0.95, 0.05]). Train a logistic regression. Show that accuracy is misleading by comparing it against F1 score, precision, and recall. - Using the breast cancer dataset (malignant = positive), sweep the decision threshold from 0.1 to 0.9 and plot both precision and recall on the same graph. Identify the threshold where they cross (this is roughly where F1 is maximized).
- On a 3-class problem (e.g., iris or load_digits), print the
classification_report. Explain the difference between the macro-average and weighted-average F1 in words, referring to the class support counts. - Intentionally introduce class imbalance by keeping only 20 samples from class 1 in the iris dataset. Compare accuracy vs. macro-average F1. Which metric better exposes the model's weakness on the minority class?
📚 Primary Sources
sklearn: Classification Metrics — comprehensive reference for every metric in this lesson.
Google ML Crash Course: Classification — excellent visual intuition for precision/recall/thresholds.