🎯 What You'll Learn
- Why linear regression fails for classification and how the sigmoid function solves the problem
- Understand the decision boundary, and how adjusting the classification threshold shifts precision vs recall
- Derive and compute log loss (binary cross-entropy) — the loss function that drives logistic regression training
- Implement binary and multi-class logistic regression with scikit-learn, including
predict_proba() - Understand the C regularization parameter and choose between L1 and L2 penalties
1 Classification vs Regression
In regression, your target variable is continuous: a house price, a temperature, a stock return. In classification, the target is a discrete category: spam or not spam, default or no default, cat or dog or bird. At first glance you might try to use linear regression for this — just predict a number and round it. But this fails in important ways.
Suppose you have a dataset where emails are labeled 0 (not spam) or 1 (spam). A linear regression model might predict 0.7 for a spam email — seems reasonable. But it could also predict 1.8 or −0.3 for extreme cases, which have no sensible interpretation as probabilities. Worse, because linear models try to fit a straight line through the data, the position of one cluster of extreme points can drag the decision boundary away from the right location. We need a model that naturally outputs values in the range [0, 1] and can be interpreted as a probability.
That is precisely what logistic regression delivers. The output P(y=1|x) is always a well-formed probability, and we apply a threshold (usually 0.5) to convert it into a class label.
Logistic regression doesn't directly predict 0 or 1. It predicts the probability that the sample belongs to class 1. You then choose a threshold to convert that probability to a hard prediction. Keeping the probability output is valuable — it lets you tune the tradeoff between false positives and false negatives for your specific business context.
2 The Sigmoid Function
The key ingredient is the sigmoid function (also called the logistic function), which squashes any real number to the open interval (0, 1):
σ(z) = 1 / (1 + e−z)
When z is a large positive number, e−z ≈ 0 and σ(z) ≈ 1. When z is a large negative number, e−z is huge and σ(z) ≈ 0. When z = 0, σ(0) = 0.5 exactly. The function has an S-shape that smoothly transitions from near-0 to near-1.
In logistic regression, z is the familiar linear combination of features: z = β₀ + β₁x₁ + β₂x₂ + ... + βₙxₙ. So the full model is:
P(y=1|x) = σ(β₀ + β₁x₁ + β₂x₂ + ... + βₙxₙ)
import numpy as np
import matplotlib.pyplot as plt
def sigmoid(z):
return 1 / (1 + np.exp(-z))
# Plot the sigmoid
z = np.linspace(-8, 8, 200)
sigma_z = sigmoid(z)
plt.figure(figsize=(8, 4))
plt.plot(z, sigma_z, 'b-', linewidth=2.5)
plt.axhline(y=0.5, color='gray', linestyle='--', alpha=0.7, label='Decision boundary (σ=0.5)')
plt.axvline(x=0, color='gray', linestyle='--', alpha=0.7)
plt.fill_between(z, sigma_z, 0.5, where=(sigma_z > 0.5), alpha=0.1, color='blue', label='Predict class 1')
plt.fill_between(z, sigma_z, 0.5, where=(sigma_z < 0.5), alpha=0.1, color='red', label='Predict class 0')
plt.xlabel('z (linear combination of features)')
plt.ylabel('σ(z) = P(y=1|x)')
plt.title('The Sigmoid Function')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
# Verify key values
print(f"σ(0) = {sigmoid(0):.4f}") # 0.5000
print(f"σ(2) = {sigmoid(2):.4f}") # 0.8808
print(f"σ(-2) = {sigmoid(-2):.4f}") # 0.1192
print(f"σ(10) = {sigmoid(10):.6f}") # 0.999955
print(f"σ(-10) = {sigmoid(-10):.6f}")# 0.000045For production code, use scipy.special.expit(z) instead of implementing sigmoid yourself. It handles numerical overflow for very large or small z values safely. sklearn's LogisticRegression uses highly optimized implementations internally.
Interactive: How w and b Shape the Sigmoid
For a single feature, z = w·x + b. The weight w controls how steep the S-curve is (and, if negative, flips which side predicts class 1), while the bias b slides the whole curve left or right. The decision boundary — the x-value where σ(z) = 0.5 — sits exactly at x = −b/w. Drag the sliders below and watch the boundary marker move along the x-axis. The dots are example points (using the email spam-score idea from this lesson: x = "spammy word count", colored by true label) — notice how some get classified correctly or incorrectly depending on where the curve currently sits.
z = 1.0·x + 0.0 — decision boundary at x = 0.00.
3 Decision Boundary and Threshold
The decision boundary is the surface in feature space where the model's predicted probability equals exactly 0.5. Everything on one side gets classified as class 1, everything on the other side as class 0. For logistic regression with the default threshold of 0.5, the decision boundary is linear — it's defined by the equation β₀ + β₁x₁ + ... + βₙxₙ = 0.
But 0.5 is just a default. You can move the threshold depending on your application's priorities. Lowering it to 0.3 means the model will predict class 1 more often — catching more true positives but also accepting more false positives. Raising it to 0.7 means the model is more conservative, predicting class 1 only when highly confident — fewer false positives but also more missed true positives.
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
import numpy as np
# Generate binary classification dataset
X, y = make_classification(n_samples=1000, n_features=2, n_redundant=0,
n_informative=2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = LogisticRegression()
model.fit(X_train, y_train)
# Get probability scores (not just labels)
proba = model.predict_proba(X_test)[:, 1] # P(y=1) for each sample
# Apply different thresholds and see effect
for threshold in [0.3, 0.5, 0.7]:
preds = (proba >= threshold).astype(int)
positives = preds.sum()
print(f"Threshold {threshold}: predicts {positives}/{len(preds)} as class 1")4 Training: Maximum Likelihood Estimation
How does logistic regression find the best parameters β? Through Maximum Likelihood Estimation (MLE). The intuition is simple: find the parameters that make the observed data as probable as possible under the model.
For a single training sample (xᵢ, yᵢ): if yᵢ=1, we want P(y=1|xᵢ) to be close to 1. If yᵢ=0, we want P(y=1|xᵢ) to be close to 0. The likelihood for one sample is:
L(β|xᵢ, yᵢ) = P(y=1|xᵢ)^yᵢ × (1 − P(y=1|xᵢ))^(1−yᵢ)
This elegantly unifies both cases: when yᵢ=1, the second term disappears; when yᵢ=0, the first term disappears. The total likelihood is the product across all training samples. Since products of small numbers underflow quickly, we take the log of the likelihood (log turns products into sums) and then negate it to get something to minimize — this gives us the log loss.
Maximizing the log-likelihood is mathematically equivalent to minimizing the log loss (binary cross-entropy). Gradient descent on the log loss finds the same parameters as MLE. This is why log loss is the natural training objective for logistic regression, not mean squared error.
5 Log Loss (Binary Cross-Entropy)
The log loss for a single sample is: L = −[y·log(p) + (1−y)·log(1−p)], where p is the predicted probability P(y=1|x). Averaged over all training samples, this is the loss function logistic regression minimises. Let's see why it has the right properties:
- If y=1 and p=0.99: loss = −log(0.99) ≈ 0.01 (near zero — we were right and confident)
- If y=1 and p=0.5: loss = −log(0.5) ≈ 0.69 (moderate — uncertain prediction)
- If y=1 and p=0.01: loss = −log(0.01) ≈ 4.61 (very high — confident and completely wrong)
The logarithm is crucial: it punishes confident wrong predictions exponentially more than uncertain ones. This is the right behavior for learning a probabilistic model.
Loss vs. predicted probability p, plotted separately for the true-label cases y=1 and y=0. Each curve blows up toward infinity as the prediction becomes confidently wrong (p→0 when y=1, or p→1 when y=0) — that steep tail is what makes log loss punish confident mistakes so much harder than MSE would.
import numpy as np
from sklearn.metrics import log_loss
# ----- Why MSE fails for classification -----
# If true label y=1 and we predict p=0.01 (confident and wrong):
y_true = 1
p = 0.01
mse = (y_true - p) ** 2 # 0.99^2 = 0.98 (seems large but bounded)
ll = -(y_true * np.log(p)) # -log(0.01) = 4.61 (much larger signal)
print(f"MSE loss for p=0.01, y=1: {mse:.4f}")
print(f"Log loss for p=0.01, y=1: {ll:.4f}")
# ---- Log loss from scratch ----
def log_loss_manual(y_true, y_pred, eps=1e-15):
"""Compute binary cross-entropy. eps prevents log(0)."""
y_pred = np.clip(y_pred, eps, 1 - eps)
return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))
y_true = np.array([1, 0, 1, 1, 0])
y_pred = np.array([0.9, 0.1, 0.8, 0.3, 0.4])
manual = log_loss_manual(y_true, y_pred)
sklearn_ll = log_loss(y_true, y_pred)
print(f"\nManual log loss: {manual:.5f}")
print(f"sklearn log_loss: {sklearn_ll:.5f}")6 Implementation with scikit-learn
sklearn's LogisticRegression handles all the gradient descent / optimization internally using solvers like LBFGS or SAG. The main parameters you'll tune are C (regularization strength) and max_iter (maximum solver iterations). Increase max_iter if you see a convergence warning.
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report, log_loss
import numpy as np
# Load a real binary classification dataset
data = load_breast_cancer()
X, y = data.data, data.target # 569 samples, 30 features; target: 0=malignant, 1=benign
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
# Always scale before logistic regression (gradient descent converges faster)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)
# Train the model
model = LogisticRegression(max_iter=1000, C=1.0, random_state=42)
model.fit(X_train_s, y_train)
# Hard predictions
y_pred = model.predict(X_test_s)
# Probability scores — shape (n_samples, n_classes)
y_proba = model.predict_proba(X_test_s)
print("Probabilities for first 5 test samples:")
print(y_proba[:5].round(3))
# Each row sums to 1.0; column 0 = P(malignant), column 1 = P(benign)
# Evaluation
print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=data.target_names))
# Log loss on test set
ll = log_loss(y_test, y_proba)
print(f"Log loss (test): {ll:.4f}")
# Model coefficients
print(f"\nIntercept: {model.intercept_}")
print(f"Top 3 most influential features (by |coefficient|):")
coef_abs = np.abs(model.coef_[0])
top3 = np.argsort(coef_abs)[-3:][::-1]
for i in top3:
print(f" {data.feature_names[i]:40s} coef={model.coef_[0][i]:.3f}")classification_report (first encounter)
That report prints three quality scores per class. Until Lesson 19 makes them precise, here's the working translation: precision — of everything the model labeled as this class, what fraction truly was? recall — of everything that truly was this class, what fraction did the model catch? f1-score — a single number balancing the two. All range 0–1, higher is better. You'll see this report in every classification lesson from now on, so this one-paragraph version will carry you until Lesson 19 gives these metrics the full treatment.
7 Multi-class Classification
Binary logistic regression handles two classes. For K > 2 classes, there are two main strategies:
One-vs-Rest (OvR)
Train K separate binary classifiers, one for each class against all others. Class k classifier asks: "Is this sample class k, or not?" At prediction time, run all K classifiers and pick the one with the highest probability. With K=5, you train 5 binary classifiers.
Softmax (Multinomial)
Train a single model with K output nodes. Use the softmax function to convert K raw scores into a proper probability distribution (all outputs between 0 and 1, sum to 1). Softmax generalises the sigmoid to multiple classes.
P(y=k|x) = exp(zₖ) / Σⱼ exp(zⱼ)
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report
# Iris dataset: 3 classes (setosa, versicolor, virginica), 4 features
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)
# ---- One-vs-Rest ----
ovr_model = LogisticRegression(multi_class='ovr', max_iter=1000, C=1.0)
ovr_model.fit(X_train_s, y_train)
ovr_pred = ovr_model.predict(X_test_s)
print("=== One-vs-Rest ===")
print(classification_report(y_test, ovr_pred, target_names=iris.target_names))
# ---- Softmax (Multinomial) ----
softmax_model = LogisticRegression(multi_class='multinomial', solver='lbfgs',
max_iter=1000, C=1.0)
softmax_model.fit(X_train_s, y_train)
softmax_pred = softmax_model.predict(X_test_s)
print("=== Softmax (Multinomial) ===")
print(classification_report(y_test, softmax_pred, target_names=iris.target_names))
# Inspect probability outputs: each row sums to 1.0
print("\nSoftmax probabilities for first 3 test samples:")
print(softmax_model.predict_proba(X_test_s[:3]).round(3))Prefer multi_class='multinomial' with solver='lbfgs' or 'saga' for most multi-class problems — it trains a single coherent model. OvR is useful when you have a very large number of classes or when you need to add a new class without retraining all the others. In sklearn ≥1.1, the default is 'auto' which picks multinomial for lbfgs.
8 Regularization: The C Parameter
Like linear regression with ridge/lasso, logistic regression can overfit on high-dimensional data. Regularization adds a penalty on large coefficients. In sklearn, this is controlled by the C parameter — the inverse of regularization strength. A small C means strong regularization; a large C means weak regularization (closer to unpenalized MLE).
- L2 (default,
penalty='l2'): Penalises the sum of squared coefficients. Shrinks all coefficients toward zero but keeps them all nonzero. Works with solvers'lbfgs','newton-cg','sag','saga'. - L1 (
penalty='l1'): Penalises the sum of absolute coefficients. Drives some coefficients to exactly zero — effectively selecting features. Requiressolver='liblinear'or'saga'. - ElasticNet (
penalty='elasticnet'): Mix of L1 and L2. Requiressolver='saga'andl1_ratioparameter.
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np
data = load_breast_cancer()
X, y = data.data, data.target
# Build pipelines with different C values
C_values = [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]
for C in C_values:
pipe = Pipeline([
('scaler', StandardScaler()),
('lr', LogisticRegression(C=C, max_iter=5000, random_state=42))
])
scores = cross_val_score(pipe, X, y, cv=5, scoring='accuracy')
print(f"C={C:6.3f} → mean accuracy = {scores.mean():.4f} ± {scores.std():.4f}")
# L1 regularization (feature selection effect)
pipe_l1 = Pipeline([
('scaler', StandardScaler()),
('lr', LogisticRegression(penalty='l1', C=0.1, solver='liblinear', max_iter=1000))
])
pipe_l1.fit(X, y)
coefs = pipe_l1.named_steps['lr'].coef_[0]
n_zero = (np.abs(coefs) < 1e-6).sum()
print(f"\nL1 with C=0.1: {n_zero}/{len(coefs)} coefficients are exactly zero")Real-World Spotlight: Email Spam Detection
Email spam detection is a canonical use case for logistic regression. Features include word counts (frequency of "free", "win", "click"), sender reputation score (0–1), number of links, whether the subject is all-caps, and HTML-to-text ratio. The target is binary: spam (1) or not spam (0).
The output is a probability — e.g., P(spam) = 0.83. The classification threshold is then a business decision: a conservative filter might use threshold = 0.9 (high precision: only flag very likely spam, never delete legitimate email). An aggressive filter might use 0.6 (high recall: catch more spam, accept occasional false positives). Companies often A/B test thresholds and monitor user complaint rates to find the right balance.
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report
import numpy as np
# Simulate a simple email spam dataset (in practice, use a real dataset like Enron)
emails = [
"Click here to win a FREE prize now!", "Meeting tomorrow at 9am in room 3",
"Congratulations! You have been selected for a cash reward", "Please review the Q3 report",
"URGENT: Your account has been compromised — verify now", "Team lunch on Friday?",
"Buy cheap meds online no prescription", "Can you join the 3pm standup?",
"FREE iPhone giveaway — limited time offer", "Budget review slides attached",
"Win win win — claim your lottery prize", "Reminder: quarterly review is next week",
]
labels = [1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0] # 1=spam, 0=ham
# Build a pipeline: TF-IDF features + Logistic Regression
pipeline = Pipeline([
('tfidf', TfidfVectorizer(ngram_range=(1, 2))),
('lr', LogisticRegression(C=1.0, max_iter=1000))
])
pipeline.fit(emails, labels)
# New emails
new_emails = ["Free money — claim your reward now!", "Updated meeting agenda for tomorrow"]
proba = pipeline.predict_proba(new_emails)[:, 1]
for email, p in zip(new_emails, proba):
verdict = "SPAM" if p >= 0.5 else "ham"
print(f"P(spam)={p:.3f} [{verdict}] {email[:50]}")Quick Check
✍️ Practice Exercises
- Load the
load_breast_cancer()dataset. Train aLogisticRegressionmodel with default settings. Print theclassification_report. Then try C=0.001 and C=100 — how does accuracy change? - Manually implement the sigmoid function and log loss from scratch using NumPy. Verify your results match
sklearn.metrics.log_losson a small example. - Using the Iris dataset (3 classes), compare OvR and multinomial logistic regression. Print both
classification_reports and discuss which performs better. - Using the breast cancer dataset, sweep the decision threshold from 0.1 to 0.9 in steps of 0.1. For each threshold, print the number of false negatives (missed cancers). Which threshold minimises false negatives?
- Try L1 regularization with C=0.01 on the breast cancer dataset. Print which features have a coefficient of exactly 0. These are the features the model has dropped.
▶ Hints
# Threshold sweep hint
y_proba = model.predict_proba(X_test_s)[:, 1]
for t in np.arange(0.1, 1.0, 0.1):
y_pred_t = (y_proba >= t).astype(int)
fn = ((y_test == 1) & (y_pred_t == 0)).sum()
print(f"threshold={t:.1f} FN={fn}")
# L1 zero-coefficient hint
lr_l1 = LogisticRegression(penalty='l1', C=0.01, solver='liblinear', max_iter=5000)
lr_l1.fit(X_train_s, y_train)
zero_features = [data.feature_names[i] for i, c in enumerate(lr_l1.coef_[0]) if abs(c) < 1e-6]
print("Dropped features:", zero_features)📚 Primary Sources
sklearn: Logistic Regression — comprehensive documentation with solver and penalty details.
Google ML Crash Course: Logistic Regression — visual walkthrough of log loss and sigmoid.