🎯 What You'll Learn
- Distinguish parameters (learned from data) from hyperparameters (set before training)
- Perform exhaustive search with
GridSearchCVand understand its computational cost - Use
RandomizedSearchCVwith continuous distributions to explore large hyperparameter spaces efficiently - Select the right evaluation metric and the right algorithm for different problem types
- Save and load tuned models with
joblibfor deployment
1 Parameters vs Hyperparameters
This distinction is fundamental and often confused:
- Parameters are learned from the training data during the fitting process. In logistic regression, the coefficients β₀, β₁, … βₙ are parameters. In a neural network, the weights and biases are parameters. You never set these manually — the optimizer finds them.
- Hyperparameters are configuration choices you set before training begins. They govern the learning process but are not themselves learned. Examples:
max_depth,learning_rate,C,n_estimators,alpha.
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
import xgboost as xgb
# Parameters vs Hyperparameters examples
print("Hyperparameters (you set these before fitting):")
print("""
LogisticRegression:
- C (regularization inverse strength)
- penalty ('l1', 'l2', 'elasticnet')
- solver, max_iter
DecisionTreeClassifier:
- max_depth, min_samples_split, min_samples_leaf
- criterion ('gini' or 'entropy')
RandomForestClassifier:
- n_estimators, max_features, max_depth
XGBClassifier:
- n_estimators, learning_rate, max_depth
- subsample, colsample_bytree, reg_alpha, reg_lambda
""")
print("Parameters (learned during fit):")
print("""
LogisticRegression.coef_ (coefficients β)
LogisticRegression.intercept_ (bias term β₀)
DecisionTree.tree_.threshold (split thresholds at each node)
RandomForest: (each tree's split thresholds)
""")
# You can inspect hyperparameters with get_params()
lr = LogisticRegression(C=0.1, penalty='l2', max_iter=1000)
print("LogisticRegression hyperparameters:")
for key, val in lr.get_params().items():
print(f" {key}: {val}")The default hyperparameters in scikit-learn were chosen to work reasonably well for many problems — but "reasonably well" is not the same as "best for your specific problem." A properly tuned model can easily outperform a default model by 5–20% on many real-world tasks. Hyperparameter tuning is the last systematic step before deployment that often yields the biggest accuracy gains for the least engineering effort.
2 GridSearchCV: Exhaustive Search
GridSearchCV tries every combination of the specified hyperparameter values, evaluates each combination using cross-validation, and returns the best combination. It is exhaustive — it guarantees finding the best configuration within the grid you define. The cost: it grows multiplicatively with the number of parameters and values.
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.metrics import roc_auc_score
import numpy as np
X_bc, y_bc = load_breast_cancer(return_X_y=True)
X_btr, X_bte, y_btr, y_bte = train_test_split(
X_bc, y_bc, test_size=0.2, stratify=y_bc, random_state=42)
# Define the grid
param_grid = {
'n_estimators': [50, 100, 200],
'max_depth': [None, 5, 10],
'min_samples_leaf': [1, 5, 10],
'max_features': ['sqrt', 'log2'],
}
# Total combinations: 3 × 3 × 3 × 2 = 54
# With cv=5: 54 × 5 = 270 model fits
print(f"Grid size: {3*3*3*2} combinations × 5 folds = {3*3*3*2*5} model fits")
import time
t0 = time.time()
grid_search = GridSearchCV(
estimator=RandomForestClassifier(n_jobs=-1, random_state=42),
param_grid=param_grid,
cv=5,
scoring='roc_auc',
n_jobs=-1, # parallelize across all available CPU cores
verbose=1,
return_train_score=True
)
grid_search.fit(X_btr, y_btr)
print(f"\nGrid search completed in {time.time()-t0:.1f}s")
print(f"\nBest parameters: {grid_search.best_params_}")
print(f"Best CV AUC: {grid_search.best_score_:.4f}")
# Evaluate on held-out test set
best_rf = grid_search.best_estimator_
test_auc = roc_auc_score(y_bte, best_rf.predict_proba(X_bte)[:, 1])
print(f"Test AUC: {test_auc:.4f}")
# Inspect all results
import pandas as pd
results_df = pd.DataFrame(grid_search.cv_results_)
top5 = results_df.nlargest(5, 'mean_test_score')[
['param_n_estimators', 'param_max_depth', 'param_min_samples_leaf',
'mean_test_score', 'std_test_score']]
print(f"\nTop 5 configurations:")
print(top5.to_string(index=False))Adding one more hyperparameter with 5 values to a grid that already has 270 fits → 270 × 5 = 1,350 fits. Adding another → 6,750. This exponential growth is the "curse of dimensionality" applied to hyperparameter search. Once you have more than 3–4 hyperparameters to tune, switch to RandomizedSearchCV. GridSearch is best for a final narrow refinement after RandomizedSearch has identified promising regions.
Holding min_samples_leaf=1 and max_features='sqrt' fixed at their winning values, here is the full 2D slice of CV results across the other two hyperparameters in the grid above — this is exactly the kind of grid you'd inspect from grid_search.cv_results_ to understand why a particular combination won:
Cross-validated ROC-AUC for every (n_estimators, max_depth) combination in the grid, with min_samples_leaf=1 and max_features='sqrt' fixed. Brighter cells = higher CV score.
3 RandomizedSearchCV: Random Sampling
Instead of trying every combination, RandomizedSearchCV samples n_iter random configurations from the distribution you specify. You can use discrete lists (like GridSearch) or continuous distributions from scipy.stats. The key insight: with many hyperparameters, most of the performance improvement comes from getting a few key parameters right — and random search finds these more efficiently than an exhaustive grid.
from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from scipy.stats import randint, uniform, loguniform
import numpy as np
import time
# Define distributions — not just discrete lists!
param_distributions = {
'n_estimators': randint(50, 500), # uniform integer in [50, 500)
'max_depth': [None, 3, 5, 7, 10, 15], # list is fine too
'min_samples_split': randint(2, 30), # uniform integer
'min_samples_leaf': randint(1, 20), # uniform integer
'max_features': ['sqrt', 'log2', 0.3, 0.5, 0.8],
'bootstrap': [True, False],
}
# n_iter=50 means 50 randomly sampled combinations
# vs 6*5*30*20*5*2 = 90,000 GridSearch combinations!
t0 = time.time()
rand_search = RandomizedSearchCV(
estimator=RandomForestClassifier(n_jobs=-1, random_state=42),
param_distributions=param_distributions,
n_iter=50, # number of random combinations to try
cv=5,
scoring='roc_auc',
n_jobs=-1,
random_state=42,
verbose=1,
return_train_score=True
)
rand_search.fit(X_btr, y_btr)
print(f"Random search completed in {time.time()-t0:.1f}s")
print(f"\nBest parameters: {rand_search.best_params_}")
print(f"Best CV AUC: {rand_search.best_score_:.4f}")
best_rand = rand_search.best_estimator_
test_auc_rand = roc_auc_score(y_bte, best_rand.predict_proba(X_bte)[:, 1])
print(f"Test AUC: {test_auc_rand:.4f}")
# loguniform is essential for hyperparameters spanning orders of magnitude
# e.g., learning_rate should be searched on a log scale
from scipy.stats import loguniform
xgb_distributions = {
'n_estimators': randint(100, 1000),
'learning_rate': loguniform(0.005, 0.3), # log-uniform in [0.005, 0.3]
'max_depth': randint(3, 10),
'subsample': uniform(0.5, 0.5), # uniform in [0.5, 1.0]
'colsample_bytree': uniform(0.5, 0.5),
'reg_alpha': loguniform(1e-4, 10),
'reg_lambda': loguniform(0.1, 100),
}
print(f"\nXGBoost param space size (with log-uniform): effectively continuous")
print("Searching 50 random configurations covers this space well")Why does random sampling beat a fixed grid for the same evaluation budget? Imagine only one of your two hyperparameters actually affects the score much — a common real situation (e.g. n_estimators matters a lot, min_samples_leaf barely matters once it's small). A 5×5 grid spends its 25 trials on just 5 distinct values of the important parameter. Random search with 25 trials tries 25 different values of the important parameter, so it has a much better chance of landing near the optimum:
25 GridSearch points fall on only 5 distinct values of each axis. Background shading shows true score depends almost entirely on n_estimators (vertical bands) — most grid points along a column are redundant.
4 Best Practices for Hyperparameter Tuning
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
import numpy as np
X_bc, y_bc = load_breast_cancer(return_X_y=True)
# ── Best Practice 1: Always include preprocessing in Pipeline ──
# This prevents data leakage — scaler fits only on training fold data
pipe = Pipeline([
('scaler', StandardScaler()),
('lr', LogisticRegression(max_iter=1000, random_state=42))
])
# Hyperparameter names in the pipeline: 'step__param'
param_grid_pipe = {
'lr__C': [0.001, 0.01, 0.1, 1, 10, 100],
'lr__penalty': ['l1', 'l2'],
'lr__solver': ['liblinear'], # supports both l1 and l2
}
grid_pipe = GridSearchCV(
pipe, param_grid_pipe,
cv=5, scoring='roc_auc', n_jobs=-1
)
grid_pipe.fit(X_bc, y_bc)
print(f"Pipeline GridSearch best: {grid_pipe.best_params_}")
print(f"CV AUC: {grid_pipe.best_score_:.4f}")
# ── Best Practice 2: Use StratifiedKFold for classification ──
from sklearn.model_selection import StratifiedKFold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# Pass cv=skf to any GridSearchCV or RandomizedSearchCV
# ── Best Practice 3: Use correct scoring for your problem ──
# Binary classification with balance: roc_auc
# Binary with imbalance: average_precision
# Regression: neg_root_mean_squared_error or r2
# Multi-class: f1_weighted
# ── Best Practice 4: Keep a held-out test set ──
from sklearn.model_selection import train_test_split
X_btr, X_bte, y_btr, y_bte = train_test_split(
X_bc, y_bc, test_size=0.15, stratify=y_bc, random_state=42
)
grid_pipe.fit(X_btr, y_btr)
print(f"\nHeld-out test AUC (never seen during tuning):")
from sklearn.metrics import roc_auc_score
test_auc_held = roc_auc_score(y_bte, grid_pipe.best_estimator_.predict_proba(X_bte)[:, 1])
print(f" {test_auc_held:.4f}")
# This is your honest estimate of production performanceStage 1 — Coarse: RandomizedSearchCV with wide ranges and n_iter=50. Identifies promising regions of the hyperparameter space quickly. Stage 2 — Fine: GridSearchCV with narrow ranges around the best values found in Stage 1. Refines the solution within the promising region. This two-stage approach typically finds near-optimal configurations 10–100× faster than GridSearch alone on the full space.
5 Selecting the Right Evaluation Metric
The most important tuning decision isn't a hyperparameter — it's choosing the right scoring metric. This directly determines what "best" means for your model.
| Problem type | Recommended metric(s) | sklearn scoring string |
|---|---|---|
| Regression | RMSE (interpretable), MAE (robust to outliers), R² | 'neg_root_mean_squared_error', 'r2' |
| Binary classification (balanced) | AUC-ROC, Accuracy, F1 | 'roc_auc', 'f1' |
| Binary classification (imbalanced) | AUC-ROC, Average Precision (AUC-PR), F1 (minority) | 'average_precision', 'roc_auc' |
| Multi-class | Weighted F1, Macro F1, AUC (OvR) | 'f1_weighted', 'roc_auc_ovr' |
| Custom business metric | Write a custom scorer | make_scorer(my_function) |
from sklearn.metrics import make_scorer, fbeta_score
import numpy as np
# Custom scorer: F2 score (recall-weighted; F1 is beta=1, F2 weights recall 2x more)
f2_scorer = make_scorer(fbeta_score, beta=2)
# Example: use in GridSearchCV
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
grid = GridSearchCV(
LogisticRegression(max_iter=1000),
param_grid={'C': [0.01, 0.1, 1, 10]},
scoring=f2_scorer, # optimize for recall-weighted F2
cv=5
)
# Use F2 when missing a positive (FN) is worse than a false alarm (FP)
# e.g., in medical screening: missing disease is worse than unnecessary follow-up
# Multiple metrics simultaneously (verbose evaluation)
from sklearn.model_selection import cross_validate
results = cross_validate(
LogisticRegression(max_iter=1000), X_bc, y_bc,
cv=5,
scoring={
'auc': 'roc_auc',
'f1': 'f1',
'precision': 'precision',
'recall': 'recall',
}
)
for metric, scores in results.items():
if metric.startswith('test_'):
name = metric.replace('test_', '')
print(f" {name:10s}: {scores.mean():.4f} ± {scores.std():.4f}")6 Model Selection Guide
Hyperparameter tuning is pointless if you start with the wrong algorithm family. Here is a practical guide based on problem characteristics:
| Algorithm | Best when | Avoid when | Tuning effort |
|---|---|---|---|
| Linear/Logistic Regression | Baseline, interpretable, <50 features | Complex nonlinear patterns | Low (just C/alpha) |
| Ridge / Lasso / ElasticNet | Many features, regularization needed | Tree-type interactions needed | Low (CV for alpha) |
| Decision Tree | Interpretable rules required | Accuracy is primary goal | Low |
| Random Forest | Robust baseline, feature importances, less tuning | Need maximum accuracy on tabular | Medium |
| XGBoost / LightGBM | Best accuracy on tabular data, large datasets | Very small datasets (<500 samples) | High |
| KNN | Small datasets, recommendation, anomaly detection | Large datasets (too slow), high dimensions | Low (just k) |
| SVM | High-dimensional text, small-medium datasets | Large datasets (memory-intensive) | Medium |
| Naive Bayes | Text classification, streaming, very fast | When feature independence is clearly violated | Very low |
7 Practical Tuning Workflow
import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.model_selection import (RandomizedSearchCV, GridSearchCV,
StratifiedKFold, train_test_split)
from sklearn.metrics import roc_auc_score
from sklearn.datasets import load_breast_cancer
from scipy.stats import randint, loguniform, uniform
X_bc, y_bc = load_breast_cancer(return_X_y=True)
X_btr, X_bte, y_btr, y_bte = train_test_split(
X_bc, y_bc, test_size=0.15, stratify=y_bc, random_state=42)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# ── Step 1: Baseline with default hyperparameters ──
baseline = xgb.XGBClassifier(n_estimators=100, n_jobs=-1, random_state=42)
baseline.fit(X_btr, y_btr)
baseline_auc = roc_auc_score(y_bte, baseline.predict_proba(X_bte)[:, 1])
print(f"Step 1 — Baseline AUC: {baseline_auc:.4f}")
# ── Step 2: Coarse RandomizedSearchCV ──
coarse_params = {
'n_estimators': randint(100, 800),
'learning_rate': loguniform(0.01, 0.3),
'max_depth': randint(3, 10),
'subsample': uniform(0.5, 0.5),
'colsample_bytree': uniform(0.5, 0.5),
'reg_alpha': loguniform(1e-4, 10),
'reg_lambda': loguniform(0.1, 100),
'min_child_weight': randint(1, 10),
}
coarse_search = RandomizedSearchCV(
xgb.XGBClassifier(n_jobs=-1, random_state=42),
coarse_params, n_iter=50, cv=skf,
scoring='roc_auc', n_jobs=-1, random_state=42, verbose=0
)
coarse_search.fit(X_btr, y_btr)
coarse_auc = roc_auc_score(y_bte, coarse_search.best_estimator_.predict_proba(X_bte)[:, 1])
print(f"Step 2 — After RandomizedSearch: CV={coarse_search.best_score_:.4f} Test={coarse_auc:.4f}")
print(f"Best params: {coarse_search.best_params_}")
# ── Step 3: Fine-tune around the best values found ──
best = coarse_search.best_params_
fine_grid = {
'n_estimators': [max(50, best['n_estimators']-100), best['n_estimators'],
best['n_estimators']+100],
'learning_rate': [best['learning_rate'] * 0.5, best['learning_rate'],
best['learning_rate'] * 2],
'max_depth': [max(2, best['max_depth']-1), best['max_depth'],
min(12, best['max_depth']+1)],
'subsample': [max(0.5, best['subsample']-0.1), best['subsample'],
min(1.0, best['subsample']+0.1)],
'colsample_bytree': [best['colsample_bytree']],
'reg_alpha': [best['reg_alpha']],
'reg_lambda': [best['reg_lambda']],
'min_child_weight': [best['min_child_weight']],
}
fine_search = GridSearchCV(
xgb.XGBClassifier(n_jobs=-1, random_state=42),
fine_grid, cv=skf, scoring='roc_auc', n_jobs=-1, verbose=0
)
fine_search.fit(X_btr, y_btr)
fine_auc = roc_auc_score(y_bte, fine_search.best_estimator_.predict_proba(X_bte)[:, 1])
print(f"\nStep 3 — After fine GridSearch: CV={fine_search.best_score_:.4f} Test={fine_auc:.4f}")
print(f"\nImprovement over baseline: {(fine_auc - baseline_auc)*100:+.2f}%")8 Saving & Loading Tuned Models
After tuning, save the complete model (including any preprocessing Pipeline) to disk for deployment. Versioning with timestamps prevents accidental overwrites and enables rollback.
import joblib
import os
from datetime import datetime
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
import xgboost as xgb
import numpy as np
# ── Saving the best model ──
best_model = fine_search.best_estimator_
# Option 1: Save estimator alone
joblib.dump(best_model, 'xgb_fraud_v1.pkl')
print(f"Model saved: {os.path.getsize('xgb_fraud_v1.pkl') / 1024:.1f} KB")
# Option 2: Save the full Pipeline (preprocessing + model) — RECOMMENDED
full_pipe = Pipeline([
('scaler', StandardScaler()),
('model', fine_search.best_estimator_)
])
full_pipe.fit(X_btr, y_btr)
# Version with timestamp
timestamp = datetime.now().strftime('%Y%m%d_%H%M')
model_path = f'xgb_fraud_pipeline_{timestamp}.pkl'
joblib.dump(full_pipe, model_path)
print(f"Pipeline saved: {model_path} ({os.path.getsize(model_path)/1024:.1f} KB)")
# ── Loading and using the model ──
loaded_pipe = joblib.load(model_path)
probabilities = loaded_pipe.predict_proba(X_bte)[:, 1]
predictions = loaded_pipe.predict(X_bte)
from sklearn.metrics import roc_auc_score
print(f"Loaded model AUC: {roc_auc_score(y_bte, probabilities):.4f}")
# ── Save metadata alongside the model ──
import json
metadata = {
'model_type': 'XGBClassifier',
'train_date': timestamp,
'cv_auc': float(fine_search.best_score_),
'test_auc': float(roc_auc_score(y_bte, probabilities)),
'best_params': fine_search.best_params_,
'n_train': len(X_btr),
'n_features': X_btr.shape[1],
'sklearn_version': '1.3.0', # record for reproducibility
}
with open(model_path.replace('.pkl', '_metadata.json'), 'w') as f:
json.dump(metadata, f, indent=2, default=str)
print(f"Metadata saved to {model_path.replace('.pkl', '_metadata.json')}")If your model was trained on scaled features, you must apply the same scaling at inference time. If you only save the model (not the scaler), you'll need to reconstruct and refit the preprocessing at prediction time — which is error-prone. Saving a Pipeline that includes all preprocessing steps ensures the same transformations are applied automatically when you call loaded_pipe.predict(X_new).
Real-World Spotlight: Optimizing a Fraud Detection Model
A payments company has a baseline XGBoost fraud detection model (AUC=0.87). The data science team runs a structured tuning campaign to improve it before the next quarterly release, following the complete two-stage workflow.
import numpy as np
import pandas as pd
import xgboost as xgb
import joblib
import json
from scipy.stats import randint, loguniform, uniform
from sklearn.model_selection import (RandomizedSearchCV, GridSearchCV,
StratifiedKFold, train_test_split)
from sklearn.metrics import (roc_auc_score, average_precision_score,
classification_report, f1_score)
from datetime import datetime
np.random.seed(42)
n = 150_000
n_fraud = int(n * 0.005) # 0.5% fraud
# Synthetic transaction data with realistic signal
X_fraud_demo = np.random.randn(n, 15)
fraud_idx = np.random.choice(n, n_fraud, replace=False)
X_fraud_demo[fraud_idx, :5] += 1.2 # fraud signal in 5 features
y_fraud_demo = np.zeros(n)
y_fraud_demo[fraud_idx] = 1
# Three-way split: train / val / test
Xtr_f, Xte_f, ytr_f, yte_f = train_test_split(
X_fraud_demo, y_fraud_demo, test_size=0.15, stratify=y_fraud_demo, random_state=42)
Xtr_f2, Xval_f, ytr_f2, yval_f = train_test_split(
Xtr_f, ytr_f, test_size=0.15, stratify=ytr_f, random_state=42)
print(f"Train: {len(Xtr_f2):,} Val: {len(Xval_f):,} Test: {len(Xte_f):,}")
print(f"Fraud rate — train: {ytr_f2.mean():.3%}")
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scale_w = (ytr_f == 0).sum() / (ytr_f == 1).sum()
# ── Step 1: Baseline ──
baseline_xgb = xgb.XGBClassifier(
n_estimators=100, scale_pos_weight=scale_w,
n_jobs=-1, random_state=42)
baseline_xgb.fit(Xtr_f, ytr_f)
baseline_auc = roc_auc_score(yte_f, baseline_xgb.predict_proba(Xte_f)[:, 1])
baseline_ap = average_precision_score(yte_f, baseline_xgb.predict_proba(Xte_f)[:, 1])
print(f"\nBaseline XGBoost: AUC={baseline_auc:.4f} AP={baseline_ap:.4f}")
# ── Step 2: RandomizedSearch ──
rand_params = {
'n_estimators': randint(100, 600),
'learning_rate': loguniform(0.01, 0.3),
'max_depth': randint(3, 9),
'subsample': uniform(0.5, 0.5),
'colsample_bytree': uniform(0.5, 0.5),
'reg_alpha': loguniform(1e-3, 10),
'reg_lambda': loguniform(0.1, 50),
'min_child_weight': randint(1, 20),
}
rand_search = RandomizedSearchCV(
xgb.XGBClassifier(scale_pos_weight=scale_w, n_jobs=-1, random_state=42),
rand_params, n_iter=50, cv=skf,
scoring='roc_auc', n_jobs=-1, random_state=42, verbose=0
)
rand_search.fit(Xtr_f, ytr_f)
print(f"After RandomizedSearch (50 iter): CV-AUC={rand_search.best_score_:.4f}")
# ── Step 3: Evaluate on held-out test ──
best_model = rand_search.best_estimator_
test_auc = roc_auc_score(yte_f, best_model.predict_proba(Xte_f)[:, 1])
test_ap = average_precision_score(yte_f, best_model.predict_proba(Xte_f)[:, 1])
print(f"Final test AUC={test_auc:.4f} AP={test_ap:.4f}")
print(f"Improvement: AUC +{(test_auc - baseline_auc)*100:+.2f}% AP +{(test_ap - baseline_ap)*100:+.2f}%")
# ── Step 4: Retrain on full train+val data before deployment ──
best_params = rand_search.best_params_.copy()
final_model = xgb.XGBClassifier(**best_params, scale_pos_weight=scale_w,
n_jobs=-1, random_state=42)
final_model.fit(Xtr_f, ytr_f) # full training set
# ── Step 5: Save ──
timestamp = datetime.now().strftime('%Y%m%d_%H%M')
joblib.dump(final_model, f'fraud_model_{timestamp}.pkl')
print(f"\nModel saved: fraud_model_{timestamp}.pkl")The two-stage tuning process improved AUC from 0.87 (baseline defaults) to ~0.93 (tuned XGBoost) — a 6-point lift representing millions of dollars in fraud prevention at scale. The final model is saved with its metadata and retraining date. In production, this model is retrained monthly with fresh data; performance is monitored weekly via a shadow deployment comparing the new vs old model on live transactions.
✍️ Practice Exercises
- Run
GridSearchCVon aRandomForestClassifierfor the Wine dataset. Try at least 3 hyperparameters with 3 values each. How many model fits does this require with 5-fold CV? - Replace the GridSearch from Exercise 1 with
RandomizedSearchCV(n_iter=20)using continuous distributions fromscipy.stats. Does it find a similar or better solution in fewer evaluations? - Build a complete Pipeline (
StandardScaler + LogisticRegression) and runGridSearchCVon it. Make sure to prefix hyperparameter names with the step name (e.g.,'lr__C'). Verify the pipeline prevents data leakage. - Save your best model from Exercise 2 using
joblib.dump. Load it back and verify that predictions are identical to the original. What happens if you forget to include the preprocessing in the saved pipeline?
📚 Primary Source for This Lesson
scikit-learn: Tuning Hyperparameters
The official guide covers GridSearchCV, RandomizedSearchCV, successive halving, and best practices with excellent code examples. Also highly recommended: Bergstra & Bengio (2012) "Random Search for Hyper-Parameter Optimization" (JMLR 13:281–305) — the paper that proved random search finds better configurations than grid search in fewer evaluations, with clear theoretical justification.