🎯 What You'll Learn
- Why large model coefficients cause overfitting and how regularization fixes it by connecting to the bias-variance tradeoff
- Understand Ridge (L2) and Lasso (L1) penalties mathematically and intuitively
- Use
Ridge,Lasso,ElasticNet,RidgeCV, andLassoCVin scikit-learn - Apply regularization to Logistic Regression with
penalty='l1'and the C hyperparameter - Visualize coefficient paths and choose the right regularizer for your problem
1 Why Regularization?
Imagine fitting a degree-15 polynomial regression to 25 training points. The model threads perfectly through every point — zero training error — yet fails catastrophically on new data. This is overfitting: the model has memorized the specific noise in the training set rather than the underlying signal.
The root cause is often large coefficients. A model with huge weights is extremely sensitive to small changes in inputs. Nudge one feature value by 0.01 and the prediction can swing wildly. Such a model has over-specialized to quirks of the training data.
Regularization adds a penalty term to the cost function that discourages large weights. Instead of minimizing just the prediction error, we minimize:
import numpy as np
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import Ridge
from sklearn.pipeline import Pipeline
np.random.seed(42)
X_train = np.sort(np.random.uniform(0, 1, 20))
y_train = np.sin(2 * np.pi * X_train) + np.random.normal(0, 0.3, 20)
# No regularization → huge coefficients
pipe_overfit = Pipeline([
('poly', PolynomialFeatures(degree=15)),
('scaler', StandardScaler()),
('ridge', Ridge(alpha=0.0)) # alpha=0 means no penalty
])
pipe_overfit.fit(X_train.reshape(-1, 1), y_train)
coefs_no_reg = pipe_overfit.named_steps['ridge'].coef_
print(f"Max |coef| WITHOUT regularization: {np.max(np.abs(coefs_no_reg)):.1f}")
# Max |coef| WITHOUT regularization: 318.4
# Ridge regularization → coefficients controlled
pipe_reg = Pipeline([
('poly', PolynomialFeatures(degree=15)),
('scaler', StandardScaler()),
('ridge', Ridge(alpha=1.0))
])
pipe_reg.fit(X_train.reshape(-1, 1), y_train)
coefs_reg = pipe_reg.named_steps['ridge'].coef_
print(f"Max |coef| WITH Ridge(alpha=1): {np.max(np.abs(coefs_reg)):.1f}")
# Max |coef| WITH Ridge(alpha=1): 1.2 ← dramatically smaller
This connects directly to the bias-variance tradeoff. No regularization → low bias, high variance (overfit). Too much regularization → high bias, low variance (underfit). The goal is to tune the regularization strength to find the generalization sweet spot.
Plain linear regression minimises: Cost = MSE(y, ŷ). Regularized regression minimises: Cost = MSE(y, ŷ) + λ × penalty(β). The λ (lambda) hyperparameter controls regularization strength. In scikit-learn, λ is called alpha. Larger alpha → stronger penalty → smaller coefficients → simpler model.
2 L2 Regularization: Ridge Regression
Ridge adds the sum of squared coefficients as the penalty:
Cost = MSE + α · Σβᵢ²
The squared term has a smooth, curved geometry. Its gradient is 2αβᵢ, which is zero only at βᵢ = 0 exactly. But in practice the MSE gradient pulls coefficients away from zero, so Ridge finds a compromise: it shrinks all coefficients towards zero proportionally, but never forces any to exactly zero. Every feature stays in the model, just with a smaller contribution.
Below is the lesson's own 500-sample, 20-feature dataset (true_coefs = [3.0, -2.0, 1.5, 0.8, -1.2, 0, 0, ..., 0] — only the first 5 features carry real signal). Each line tracks one feature's coefficient as λ sweeps from nearly 0 to 10. Toggle between Ridge and Lasso, then drag the λ slider to see exactly how the two penalties diverge.
Ridge, λ = 0.062 — all 20 coefficients are nonzero, shrinking smoothly together.
Coefficient values at the current λ — thick/colored bars are the 5 true signal features, faint bars are the 15 noise features.
Count of nonzero coefficients across the full λ range.
import numpy as np
from sklearn.linear_model import Ridge, RidgeCV
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error
np.random.seed(42)
n, p = 500, 20
X = np.random.randn(n, p)
# True signal: only first 5 features matter
true_coefs = np.array([3.0, -2.0, 1.5, 0.8, -1.2] + [0.0]*15)
y = X @ true_coefs + np.random.randn(n) * 0.5
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)
print(f"{'alpha':>8} {'Test MAE':>10} {'Max coef':>10} {'Non-zero':>10}")
print("-" * 50)
for alpha in [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]:
ridge = Ridge(alpha=alpha)
ridge.fit(X_train_s, y_train)
mae = mean_absolute_error(y_test, ridge.predict(X_test_s))
max_c = np.max(np.abs(ridge.coef_))
nonzero = np.sum(np.abs(ridge.coef_) > 1e-4)
print(f"{alpha:>8.3f} {mae:>10.4f} {max_c:>10.3f} {nonzero:>10}")
# alpha=0.001 Test MAE=0.0447 Max coef=2.987 Non-zero=20
# alpha=1.000 Test MAE=0.0478 Max coef=2.481 Non-zero=20
# alpha=100.0 Test MAE=0.0873 Max coef=0.432 Non-zero=20
# Non-zero stays at 20 for ALL alphas — Ridge NEVER zeroes out features
Both penalties apply to raw coefficient magnitudes. If feature A has values in [0, 0.01] and feature B in [0, 1000], the raw coefficient for A will be 100,000× larger just to have equal effect — so the penalty unfairly targets A. Always apply StandardScaler first (or use sklearn's Pipeline) to put all features on the same scale.
3 L1 Regularization: Lasso Regression
Lasso (Least Absolute Shrinkage and Selection Operator) uses the sum of absolute values:
Cost = MSE + α · Σ|βᵢ|
The absolute value function has a sharp corner at zero — geometrically, the L1 ball is a diamond. At any corner of this diamond (where one coefficient is zero), the subdifferential includes zero, meaning the optimizer can legally "stop" exactly at a corner. This is why Lasso sets coefficients to exactly zero, removing features from the model entirely. As you increase α, more and more features are zeroed out, giving you automatic feature selection.
The dashed ellipses are contours of equal MSE (the unconstrained loss); the solid shape is the regularization constraint region. Where the smallest-loss ellipse first touches the constraint shape is the regularized solution. The diamond's sharp corners sit on the axes — so the contour often touches a corner first, zeroing a coefficient. The circle has no corners, so the touch point is almost always off-axis, leaving every coefficient nonzero but shrunk.
from sklearn.linear_model import Lasso
from sklearn.metrics import mean_absolute_error
import numpy as np
# Same 500-sample, 20-feature dataset; only first 5 are real signals
feature_names = [f'feature_{i:02d}' for i in range(20)]
print(f"{'alpha':>7} {'MAE':>8} {'# kept':>7} {'Surviving features'}")
print("-" * 70)
for alpha in [0.001, 0.01, 0.05, 0.10, 0.50, 1.00]:
lasso = Lasso(alpha=alpha, max_iter=5000)
lasso.fit(X_train_s, y_train)
surviving = [feature_names[i] for i in range(20) if abs(lasso.coef_[i]) > 1e-4]
mae = mean_absolute_error(y_test, lasso.predict(X_test_s))
print(f"{alpha:>7.3f} {mae:>8.4f} {len(surviving):>7} {surviving}")
# alpha=0.001 MAE=0.0447 #kept=20 [all features]
# alpha=0.010 MAE=0.0448 #kept=14
# alpha=0.050 MAE=0.0452 #kept= 7
# alpha=0.100 MAE=0.0460 #kept= 5 [feature_00, feature_01, feature_02, feature_03, feature_04]
# alpha=0.500 MAE=0.0601 #kept= 2
# alpha=1.000 MAE=0.1023 #kept= 1
# At alpha=0.1, Lasso EXACTLY recovers the 5 true features!
When two features are highly correlated, Lasso tends to arbitrarily pick one and zero the other. Which one survives can change with small perturbations to training data. This is not a bug — it reflects genuine ambiguity — but Lasso-based feature selection should always be validated with cross-validation. For correlated feature groups, ElasticNet is more stable.
4 ElasticNet: Combining L1 and L2
ElasticNet blends both penalties using the l1_ratio parameter:
Cost = MSE + α · [l1_ratio · Σ|βᵢ| + (1 − l1_ratio) · Σβᵢ²]
When l1_ratio=1, ElasticNet is identical to Lasso. When l1_ratio=0, it is Ridge. In between, you get sparsity (from L1) combined with stability among correlated feature groups (from L2). ElasticNet is particularly valuable in genomics, text modeling, and any domain where you have many features with rich correlation structure.
from sklearn.linear_model import Lasso, Ridge, ElasticNet
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error
# Dataset where x1 and x2 are strongly correlated
np.random.seed(42)
n = 500
x1 = np.random.randn(n)
x2 = x1 + np.random.randn(n) * 0.2 # r ≈ 0.98 — nearly identical
x3 = np.random.randn(n)
X_corr = np.column_stack([x1, x2, x3] + [np.random.randn(n) for _ in range(17)])
y_corr = 3.0*x1 + 2.0*x3 + np.random.randn(n) * 0.5
# True model: x1 and x3 matter; x2 is a correlated copy of x1
Xc_train, Xc_test, yc_train, yc_test = train_test_split(X_corr, y_corr, test_size=0.2, random_state=42)
sc2 = StandardScaler()
Xc_tr_s = sc2.fit_transform(Xc_train)
Xc_te_s = sc2.transform(Xc_test)
print("Feature 0 (x1) and Feature 1 (x2) are correlated — how does each model handle them?")
for name, model in [
('Ridge', Ridge(alpha=1.0)),
('Lasso', Lasso(alpha=0.05)),
('ElasticNet', ElasticNet(alpha=0.05, l1_ratio=0.5)),
]:
model.fit(Xc_tr_s, yc_train)
c0, c1, c2 = model.coef_[0], model.coef_[1], model.coef_[2]
mae = mean_absolute_error(yc_test, model.predict(Xc_te_s))
print(f" {name:12s}: coef[x1]={c0:.3f} coef[x2]={c1:.3f} coef[x3]={c2:.3f} MAE={mae:.4f}")
# Ridge: coef[x1]=1.412 coef[x2]=1.378 coef[x3]=1.803 MAE=0.0521
# Lasso: coef[x1]=2.481 coef[x2]=0.000 coef[x3]=1.854 MAE=0.0503
# ElasticNet: coef[x1]=1.702 coef[x2]=0.432 coef[x3]=1.836 MAE=0.0499
# Ridge distributes weight across both; Lasso arbitrarily drops x2; ElasticNet keeps both
# l1_ratio sweep
print("\nElasticNet l1_ratio sweep (alpha=0.05):")
for l1r in [0.0, 0.25, 0.5, 0.75, 1.0]:
en = ElasticNet(alpha=0.05, l1_ratio=l1r, max_iter=5000)
en.fit(Xc_tr_s, yc_train)
nonzero = np.sum(np.abs(en.coef_) > 1e-4)
mae = mean_absolute_error(yc_test, en.predict(Xc_te_s))
print(f" l1_ratio={l1r:.2f}: kept={nonzero:2d} features MAE={mae:.4f}")
The printed results above, visualized — watch what happens to the correlated pair (x1, x2) specifically:
x1 and x2 are near-duplicates (r ≈ 0.98); x3 is an independent true signal. Ridge splits weight evenly across x1/x2. Lasso arbitrarily zeroes x2 entirely. ElasticNet keeps both x1 and x2 nonzero, splitting weight less evenly than Ridge but without discarding either.
5 Choosing α: RidgeCV and LassoCV
Manually tuning alpha is tedious and prone to test-set leakage. Scikit-learn provides built-in cross-validated versions that automatically find the best α from a set of candidates.
from sklearn.linear_model import RidgeCV, LassoCV
import numpy as np
# RidgeCV uses built-in leave-one-out CV (analytically efficient)
alphas_to_try = [0.001, 0.01, 0.1, 1, 5, 10, 50, 100, 500, 1000]
ridge_cv = RidgeCV(alphas=alphas_to_try, cv=5, scoring='neg_mean_absolute_error')
ridge_cv.fit(X_train_s, y_train)
print(f"Best alpha (Ridge): {ridge_cv.alpha_}")
print(f"Coefficients: {ridge_cv.coef_[:5].round(3)}")
# Best alpha (Ridge): 0.1
# Coefficients: [ 2.879 -1.923 1.418 0.738 -1.101]
# LassoCV computes a full regularization path efficiently via warm starts
lasso_cv = LassoCV(cv=5, max_iter=10000, random_state=42, n_alphas=100)
lasso_cv.fit(X_train_s, y_train)
print(f"\nBest alpha (Lasso): {lasso_cv.alpha_:.5f}")
selected_idx = np.where(np.abs(lasso_cv.coef_) > 1e-4)[0]
print(f"Features selected: {selected_idx}")
print(f"Their coefficients: {lasso_cv.coef_[selected_idx].round(3)}")
# Best alpha (Lasso): 0.04892
# Features selected: [0 1 2 3 4] ← exactly the 5 true features
# Their coefficients: [ 2.891 -1.912 1.428 0.752 -1.095]
RidgeCV has an analytical shortcut for leave-one-out cross-validation, making it extremely fast even with hundreds of alpha values. LassoCV uses coordinate descent with warm starts along the regularization path — each alpha value continues from the previous solution, which is much faster than fitting from scratch. Both are far faster than looping over alphas yourself with cross_val_score (or the automated grid-search tool coming in Lesson 27).
6 Regularization in Logistic Regression
Regularization applies equally to classification. Scikit-learn's LogisticRegression uses L2 regularization by default. Important: it uses the inverse convention — the hyperparameter is C = 1/λ. Small C → strong regularization. Large C → weak regularization. This can catch people out coming from linear regression.
from sklearn.linear_model import LogisticRegression, LogisticRegressionCV
from sklearn.datasets import make_classification
from sklearn.metrics import roc_auc_score
import numpy as np
X_cls, y_cls = make_classification(
n_samples=1000, n_features=20, n_informative=5,
n_redundant=10, random_state=42
)
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
X_cls_tr, X_cls_te, y_cls_tr, y_cls_te = train_test_split(
X_cls, y_cls, test_size=0.2, random_state=42)
sc3 = StandardScaler()
X_cls_tr_s = sc3.fit_transform(X_cls_tr)
X_cls_te_s = sc3.transform(X_cls_te)
print("L2 Logistic Regression (default penalty='l2'):")
for C in [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]:
lr = LogisticRegression(penalty='l2', C=C, max_iter=1000)
lr.fit(X_cls_tr_s, y_cls_tr)
auc = roc_auc_score(y_cls_te, lr.predict_proba(X_cls_te_s)[:, 1])
nonzero = np.sum(np.abs(lr.coef_[0]) > 1e-4)
print(f" C={C:6.3f} AUC={auc:.4f} nonzero features={nonzero}")
print("\nL1 Logistic Regression (produces sparse models):")
for C in [0.001, 0.01, 0.1, 1.0]:
lr = LogisticRegression(penalty='l1', C=C, solver='liblinear', max_iter=1000)
lr.fit(X_cls_tr_s, y_cls_tr)
nonzero = np.sum(np.abs(lr.coef_[0]) > 1e-4)
auc = roc_auc_score(y_cls_te, lr.predict_proba(X_cls_te_s)[:, 1])
print(f" C={C:5.3f} AUC={auc:.4f} features selected={nonzero}")
# Auto-select best C with cross-validation
lr_cv = LogisticRegressionCV(
Cs=[0.001, 0.01, 0.1, 1, 10, 100],
cv=5, penalty='l2', scoring='roc_auc',
max_iter=1000, random_state=42
)
lr_cv.fit(X_cls_tr_s, y_cls_tr)
print(f"\nBest C (LogisticRegressionCV): {lr_cv.C_[0]:.4f}")
print(f"Test AUC: {roc_auc_score(y_cls_te, lr_cv.predict_proba(X_cls_te_s)[:,1]):.4f}")
7 Visualizing Coefficient Paths
A regularization path shows how every coefficient changes as α increases from nearly zero (almost unrestricted) to very large (all near-zero). Lasso paths are particularly informative: they reveal the order in which features are "turned off" as you tighten the penalty. The last feature to turn off is the single most important predictor.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import lasso_path
from sklearn.preprocessing import StandardScaler
np.random.seed(42)
n, p = 500, 10
X_path = np.random.randn(n, p)
true_c = np.array([3.0, -2.0, 1.5, 0.8, -1.2, 0, 0, 0, 0, 0])
y_path = X_path @ true_c + np.random.randn(n) * 0.5
scaler_path = StandardScaler()
X_path_s = scaler_path.fit_transform(X_path)
# lasso_path returns decreasing alphas and coef matrix shape (p, n_alphas)
alphas, coefs, _ = lasso_path(X_path_s, y_path, n_alphas=100)
plt.figure(figsize=(10, 5))
for i in range(p):
style = dict(linewidth=2.5, alpha=0.9) if true_c[i] != 0 else dict(linewidth=1, alpha=0.35)
plt.plot(np.log10(alphas), coefs[i], **style,
label=f'Feat {i} (true={true_c[i]:.1f})' if true_c[i] != 0 else None)
plt.axvline(x=np.log10(0.1), color='red', linestyle='--', alpha=0.7, label='α = 0.1')
plt.xlabel('log₁₀(α) → stronger regularization (right)')
plt.ylabel('Coefficient value')
plt.title('Lasso Regularization Path | Thick = true signal, thin = noise')
plt.legend(fontsize=8, loc='upper left')
plt.gca().invert_xaxis()
plt.tight_layout()
plt.savefig('lasso_path.png', dpi=150, bbox_inches='tight')
# Print when each feature hits zero
for i in range(p):
zero_idx = np.where(np.abs(coefs[i]) < 1e-4)[0]
if len(zero_idx):
a0 = alphas[zero_idx[0]]
print(f"Feature {i:2d} (true={true_c[i]:+.1f}) zeroed at α={a0:.4f}")
# Feature 0 (true=+3.0) zeroed at α=0.3521 ← strongest, last to zero
# Feature 4 (true=-1.2) zeroed at α=0.0891 ← weakest true signal
# Feature 5 (true=+0.0) zeroed at α=0.0124 ← noise, eliminated early
8 When to Use Which Regularizer
| Scenario | Recommended | Why |
|---|---|---|
| Many irrelevant features | Lasso | Automatically zeros out irrelevant features, sparse model |
| Highly correlated features | Ridge or ElasticNet | Ridge distributes weight; ElasticNet keeps feature groups |
| Need interpretability | Lasso | Sparse model — only key features retain nonzero coefficients |
| All features genuinely matter | Ridge | Shrinks all evenly without discarding valid signals |
| Large p, moderate n, some noise | ElasticNet | Balances sparsity and stability; tune l1_ratio via CV |
| Unsure which to pick | ElasticNetCV | Searches both α and l1_ratio automatically with CV |
Real-World Spotlight: Customer Lifetime Value Prediction
A subscription company needs to predict Customer Lifetime Value (CLV) using 50+ features: demographics, behavioral metrics, and highly correlated marketing channel data. Many features overlap; the business team wants a sparse, interpretable model highlighting the true drivers.
import numpy as np
import pandas as pd
from sklearn.linear_model import Ridge, Lasso, ElasticNet, LassoCV
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error
np.random.seed(42)
n = 2000
# 50 features: 12 truly predictive, 38 noise/correlated noise
feature_names = [f'feature_{i:02d}' for i in range(50)]
X_clv = np.random.randn(n, 50)
# Marketing metrics (features 20-35) are correlated with each other
for i in range(20, 36):
X_clv[:, i] = X_clv[:, 20] + np.random.randn(n) * 0.4
true_coefs = np.zeros(50)
true_coefs[:12] = np.random.randn(12) * 2
y_clv = X_clv @ true_coefs + np.random.randn(n) * 5
Xclv_tr, Xclv_te, yclv_tr, yclv_te = train_test_split(
X_clv, y_clv, test_size=0.2, random_state=42)
sc_clv = StandardScaler()
Xclv_tr_s = sc_clv.fit_transform(Xclv_tr)
Xclv_te_s = sc_clv.transform(Xclv_te)
# LassoCV auto-selects alpha via cross-validation
lasso_cv = LassoCV(cv=5, max_iter=10000, random_state=42)
lasso_cv.fit(Xclv_tr_s, yclv_tr)
selected_idx = np.where(np.abs(lasso_cv.coef_) > 0.01)[0]
selected_names = [feature_names[i] for i in selected_idx]
mae_lasso = mean_absolute_error(yclv_te, lasso_cv.predict(Xclv_te_s))
print(f"LassoCV best alpha : {lasso_cv.alpha_:.5f}")
print(f"Features selected : {len(selected_idx)} out of 50")
print(f"Selected features : {selected_names}")
print(f"Test MAE (Lasso) : {mae_lasso:.3f}")
# Compare all three regularizers
for name, model in [
('Ridge', Ridge(alpha=1.0)),
('ElasticNet', ElasticNet(alpha=0.1, l1_ratio=0.5)),
]:
model.fit(Xclv_tr_s, yclv_tr)
mae = mean_absolute_error(yclv_te, model.predict(Xclv_te_s))
nonzero = np.sum(np.abs(model.coef_) > 0.01)
print(f"{name:12s}: MAE={mae:.3f} nonzero features={nonzero}")
# LassoCV best alpha : 0.04201
# Features selected : 12 out of 50 ← correctly identifies all true features!
# Test MAE (Lasso) : 4.821
# Ridge : MAE=5.012 nonzero features=50
# ElasticNet : MAE=4.903 nonzero features=18
LassoCV recovered all 12 true predictors and achieved the lowest test MAE. The business team can now present a sparse, 12-variable model rather than a 50-variable black-box. ElasticNet was a close second and handled the correlated marketing features (20–35) more gracefully by retaining some from the group rather than arbitrarily discarding all but one.
✍️ Practice Exercises
- Load the California Housing dataset (
from sklearn.datasets import fetch_california_housing). Standardize features and compareRidge,Lasso, andElasticNeton test MAE. Which regularizer performs best? - Use
LassoCV(cv=5)on California Housing. Which features survive? Do the survivors make domain sense for predicting house prices? - Create a dataset with two nearly identical features (
x2 = x1 + 0.01 * noise). Fit Ridge and Lasso separately. What happens to each pair of coefficients? - Fit
LogisticRegression(penalty='l1', C=0.1, solver='liblinear')on the Breast Cancer dataset. Which features survive? Compare test AUC to the default L2 logistic regression.
📚 Primary Source for This Lesson
scikit-learn: Linear Models — Ridge, Lasso, ElasticNet
The canonical reference for all regularized linear models, with mathematical formulations, solver options, and practical guidance. Also recommended: Hastie, Tibshirani & Friedman, The Elements of Statistical Learning (freely available online), Chapter 3 for the theoretical foundations of shrinkage methods.