🔴 ML / AI

Python + Scikit-learn: ML Pipelines

📖 Lesson 46 ⏱ 55 min 🧪 5 questions 💻 3 exercises

🎯 Learning Objectives

Understand scikit-learn's estimator API: fit, transform, predict, score
Split data correctly with train_test_split and stratified k-fold cross-validation
Build preprocessing pipelines with Pipeline and ColumnTransformer
Train and evaluate classification and regression models
Tune hyperparameters with GridSearchCV and RandomizedSearchCV
Evaluate models with appropriate metrics: accuracy, F1, ROC-AUC, RMSE, R²
Persist and load trained pipelines with joblib

1 · The Scikit-learn API

Scikit-learn provides a unified estimator interface: every model and transformer exposes fit(), and either transform() (transformers) or predict() (predictors) — or both. This consistency means you can swap algorithms with a single line change while keeping the rest of your code intact.

Install with:

pip install scikit-learn

terminal

Three estimator roles:

Role	Methods	Examples
Transformer	`fit(X)`, `transform(X)`, `fit_transform(X)`	`StandardScaler`, `OneHotEncoder`, `PCA`
Predictor	`fit(X, y)`, `predict(X)`, `score(X, y)`	`LogisticRegression`, `RandomForest`, `SVR`
Both	all of the above	`KMeans`, `FeatureAgglomeration`

from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
import numpy as np

X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([0, 0, 1, 1])

# Transformer
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)    # fit + transform in one step

# Predictor
model = LogisticRegression()
model.fit(X_scaled, y)
predictions = model.predict(X_scaled)
probabilities = model.predict_proba(X_scaled)
score = model.score(X_scaled, y)      # accuracy by default

estimator_api.py

The golden rule — fit only on training data, transform both train and test. Fitting on the full dataset causes data leakage: the model sees test-set statistics during training, producing optimistically biased evaluation metrics.

2 · Splitting & Cross-Validation

Proper data splitting ensures your evaluation reflects real-world performance. Use train_test_split for a simple hold-out and cross-validation when you need a more robust estimate.

from sklearn.model_selection import (
    train_test_split, cross_val_score,
    StratifiedKFold, KFold, cross_validate,
)
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier

X, y = load_breast_cancer(return_X_y=True)

# ── Basic split ──
X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,
    random_state=42,
    stratify=y,          # preserve class balance in both splits
)
print(X_train.shape, X_test.shape)   # (455, 30) (114, 30)

# ── Cross-validation ──
model = RandomForestClassifier(n_estimators=100, random_state=42)
cv_scores = cross_val_score(model, X, y, cv=5, scoring="accuracy")
print(f"CV accuracy: {cv_scores.mean():.3f} ± {cv_scores.std():.3f}")

# ── Stratified K-Fold (explicit, for imbalanced classes) ──
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=skf, scoring="roc_auc")

# ── cross_validate: multiple metrics at once ──
results = cross_validate(
    model, X, y, cv=5,
    scoring=["accuracy", "f1", "roc_auc"],
    return_train_score=True,
)
print(results["test_accuracy"].mean())

splitting.py

Always use stratify=y for classification tasks — without it, a random split may put most of one class into training, making evaluation unreliable.

3 · Preprocessing

Real-world data is messy: missing values, different scales, categorical features. Scikit-learn provides a rich set of transformers to handle all of these before feeding data to a model.

import numpy as np
import pandas as pd
from sklearn.preprocessing import (
    StandardScaler, MinMaxScaler, RobustScaler,
    OneHotEncoder, OrdinalEncoder, LabelEncoder,
    PolynomialFeatures, FunctionTransformer,
)
from sklearn.impute import SimpleImputer, KNNImputer

# ── Scalers ──
X = np.array([[1., 2.], [3., 4.], [5., 6.]])
StandardScaler().fit_transform(X)    # mean=0, std=1 per feature
MinMaxScaler().fit_transform(X)      # scale to [0, 1]
RobustScaler().fit_transform(X)      # uses median/IQR — robust to outliers

# ── Encoders ──
cats = np.array([["cat"], ["dog"], ["cat"], ["bird"]])
OneHotEncoder(sparse_output=False).fit_transform(cats)
# [[1,0,0], [0,0,1], [1,0,0], [0,1,0]]

OrdinalEncoder().fit_transform(np.array([["low"],["med"],["high"],["low"]]))
# [[0.], [2.], [1.], [0.]]   (alphabetical order by default)

# ── Imputers ──
X_missing = np.array([[1., np.nan], [3., 4.], [np.nan, 6.]])
SimpleImputer(strategy="mean").fit_transform(X_missing)
SimpleImputer(strategy="median").fit_transform(X_missing)
SimpleImputer(strategy="most_frequent").fit_transform(X_missing)
KNNImputer(n_neighbors=2).fit_transform(X_missing)   # impute using k nearest rows

# ── Polynomial features ──
pf = PolynomialFeatures(degree=2, include_bias=False)
pf.fit_transform(np.array([[2., 3.]]))
# [2., 3., 4., 6., 9.]  (x1, x2, x1², x1·x2, x2²)

# ── Custom transformer ──
import numpy as np
log_transform = FunctionTransformer(np.log1p)   # log(1+x), avoids log(0)

preprocessing.py

Scaler comparison — when to use each:

Scaler	Centres by	Scales by	When to use
`StandardScaler`	Mean	Std deviation	Default choice; data is roughly Gaussian
`MinMaxScaler`	Min	Range (max−min)	Need bounded [0,1] values; neural networks, image pixels
`RobustScaler`	Median	IQR (Q3−Q1)	Data has significant outliers

4 · Pipeline & ColumnTransformer

A Pipeline chains preprocessing steps and a final estimator into a single object. A ColumnTransformer lets you apply different transformations to different column subsets — essential for DataFrames with mixed numeric and categorical features.

import pandas as pd
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split

# ── Sample data with mixed types ──
df = pd.DataFrame({
    "age":      [25, np.nan, 45, 30, 50],
    "salary":   [50000, 72000, 85000, np.nan, 110000],
    "dept":     ["Eng", "Sales", None, "Eng", "Mgmt"],
    "promoted": [0, 1, 1, 0, 1],
})

X = df.drop("promoted", axis=1)
y = df["promoted"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=0)

# ── Column-specific pipelines ──
numeric_features = ["age", "salary"]
categorical_features = ["dept"]

numeric_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler",  StandardScaler()),
])

categorical_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
])

preprocessor = ColumnTransformer([
    ("num", numeric_pipeline,  numeric_features),
    ("cat", categorical_pipeline, categorical_features),
])

# ── Full pipeline: preprocessing + model ──
pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier",   GradientBoostingClassifier(n_estimators=100, random_state=42)),
])

pipeline.fit(X_train, y_train)
print(f"Test accuracy: {pipeline.score(X_test, y_test):.3f}")
predictions = pipeline.predict(X_test)

pipeline_demo.py

Why Pipeline prevents leakage: When you call pipeline.fit(X_train, y_train), it calls fit_transform() on each preprocessing step sequentially using only training data. When you call pipeline.predict(X_test), it calls transform() on each step (no refitting) — test data is transformed using training-set statistics. One call fits all steps correctly.

5 · Classification & Regression Models

Scikit-learn ships dozens of models that all share the same fit/predict/score API. Here's a quick comparison across families:

from sklearn.linear_model import LogisticRegression, Ridge, Lasso, ElasticNet
from sklearn.ensemble import (
    RandomForestClassifier, GradientBoostingClassifier,
    RandomForestRegressor, GradientBoostingRegressor,
)
from sklearn.svm import SVC, SVR
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.datasets import load_breast_cancer, load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# ── Classification ──
X, y = load_breast_cancer(return_X_y=True)
X = StandardScaler().fit_transform(X)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

models = {
    "Logistic Regression": LogisticRegression(max_iter=1000),
    "Random Forest":       RandomForestClassifier(n_estimators=100, random_state=42),
    "Gradient Boosting":   GradientBoostingClassifier(random_state=42),
    "SVM (RBF kernel)":    SVC(probability=True),
    "KNN (k=5)":           KNeighborsClassifier(n_neighbors=5),
}
for name, m in models.items():
    m.fit(X_tr, y_tr)
    print(f"{name:25s} accuracy={m.score(X_te, y_te):.4f}")

# ── Regression ──
X_r, y_r = load_diabetes(return_X_y=True)
X_r = StandardScaler().fit_transform(X_r)
X_rtr, X_rte, y_rtr, y_rte = train_test_split(X_r, y_r, test_size=0.2, random_state=42)

reg_models = {
    "Ridge (L2)":        Ridge(alpha=1.0),
    "Lasso (L1)":        Lasso(alpha=0.1),
    "ElasticNet":        ElasticNet(alpha=0.1, l1_ratio=0.5),
    "Random Forest Reg": RandomForestRegressor(n_estimators=100, random_state=42),
    "Gradient Boosting": GradientBoostingRegressor(random_state=42),
}
from sklearn.metrics import root_mean_squared_error, r2_score
for name, m in reg_models.items():
    m.fit(X_rtr, y_rtr)
    preds = m.predict(X_rte)
    print(f"{name:20s} RMSE={root_mean_squared_error(y_rte, preds):.2f}  R²={r2_score(y_rte, preds):.3f}")

models.py

Model selection guidance:

Algorithm	Type	Regularisation	Interpretability	Scalability	When to use
Logistic / Linear Regression	Linear	L1, L2, ElasticNet	High	Excellent	Baseline; interpretable coefficients needed
Decision Tree	Tree	Depth/leaf pruning	High	Good	Quick exploration; feature importance
Random Forest	Ensemble (bagging)	n_estimators, depth	Medium	Good (parallel)	Strong default; tabular data
Gradient Boosting	Ensemble (boosting)	Learning rate, depth	Low	Medium	Best accuracy on structured data
SVM	Kernel	C, gamma	Low	Poor (>10k rows)	Small/medium datasets; non-linear boundaries
KNN	Instance-based	k, distance metric	Medium	Poor at scale	Simple problems; few features

6 · Evaluation Metrics

Choosing the right metric is as important as choosing the right model. Use classification metrics for discrete labels and regression metrics for continuous targets.

from sklearn.metrics import (
    # Classification
    accuracy_score, precision_score, recall_score, f1_score,
    classification_report, confusion_matrix, roc_auc_score, roc_curve,
    # Regression
    mean_absolute_error, mean_squared_error, root_mean_squared_error,
    r2_score, mean_absolute_percentage_error,
)
import matplotlib.pyplot as plt
import numpy as np

# ── Classification metrics ──
y_true = np.array([0, 0, 1, 1, 1, 0, 1, 0])
y_pred = np.array([0, 1, 1, 1, 0, 0, 1, 1])
y_prob = np.array([0.1, 0.7, 0.9, 0.8, 0.4, 0.2, 0.85, 0.6])

print(classification_report(y_true, y_pred, target_names=["No", "Yes"]))
print("Confusion matrix:\n", confusion_matrix(y_true, y_pred))
print("ROC-AUC:", roc_auc_score(y_true, y_prob))

# ── ROC curve ──
fpr, tpr, thresholds = roc_curve(y_true, y_prob)
fig, ax = plt.subplots(figsize=(6, 5))
ax.plot(fpr, tpr, label=f"AUC={roc_auc_score(y_true, y_prob):.3f}")
ax.plot([0,1],[0,1],"--", color="gray")
ax.set_xlabel("FPR"); ax.set_ylabel("TPR"); ax.legend()
ax.set_title("ROC Curve")

# ── Regression metrics ──
y_r_true = np.array([3.0, -0.5, 2.0, 7.0])
y_r_pred = np.array([2.5,  0.0, 2.0, 8.0])
print(f"MAE:  {mean_absolute_error(y_r_true, y_r_pred):.3f}")
print(f"RMSE: {root_mean_squared_error(y_r_true, y_r_pred):.3f}")
print(f"R²:   {r2_score(y_r_true, y_r_pred):.3f}")
print(f"MAPE: {mean_absolute_percentage_error(y_r_true, y_r_pred):.3f}")

metrics.py

Metrics reference:

Metric	Formula / Idea	Range	Interpretation	When to use
Accuracy	Correct / Total	[0, 1]	Overall correctness	Balanced classes only
Precision	TP / (TP + FP)	[0, 1]	Of predicted positives, how many correct	Cost of false positives is high (spam)
Recall	TP / (TP + FN)	[0, 1]	Of actual positives, how many found	Cost of false negatives is high (disease)
F1	2·P·R / (P + R)	[0, 1]	Harmonic mean of precision & recall	Imbalanced classes
ROC-AUC	Area under ROC curve	[0, 1]	Ranking quality across all thresholds	Comparing classifiers; threshold-agnostic
MAE	Mean \|y − ŷ\|	[0, ∞)	Average absolute error	Easy interpretation; robust to outliers
RMSE	√Mean(y − ŷ)²	[0, ∞)	Penalises large errors more	Large errors are costly
R²	1 − SS_res / SS_tot	(−∞, 1]	Proportion of variance explained	Overall regression quality

7 · Hyperparameter Tuning

Models have hyperparameters (set before training) that dramatically affect performance. Scikit-learn provides two main search strategies: exhaustive grid search and randomised sampling.

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_breast_cancer
from scipy.stats import randint, uniform
import numpy as np

X, y = load_breast_cancer(return_X_y=True)

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("clf",    RandomForestClassifier(random_state=42)),
])

# ── GridSearchCV — exhaustive search ──
param_grid = {
    "clf__n_estimators": [50, 100, 200],
    "clf__max_depth":    [None, 5, 10],
    "clf__min_samples_split": [2, 5],
}
grid_search = GridSearchCV(
    pipeline, param_grid,
    cv=5, scoring="roc_auc", n_jobs=-1, verbose=1,
)
grid_search.fit(X, y)
print("Best params:", grid_search.best_params_)
print("Best AUC:  ", grid_search.best_score_)

# ── RandomizedSearchCV — sample from distributions ──
param_dist = {
    "clf__n_estimators":     randint(50, 500),
    "clf__max_depth":        [None, 5, 10, 20],
    "clf__min_samples_split": randint(2, 20),
    "clf__max_features":     uniform(0.1, 0.9),
}
random_search = RandomizedSearchCV(
    pipeline, param_dist,
    n_iter=50, cv=5, scoring="roc_auc",
    random_state=42, n_jobs=-1,
)
random_search.fit(X, y)
print("Best params:", random_search.best_params_)

tuning.py

Use GridSearchCV when the search space is small (< ~100 combinations) and RandomizedSearchCV for large spaces — 50 random trials often finds near-optimal parameters faster than exhaustive grid search. Set n_jobs=-1 to use all CPU cores.

Feature Importance & Interpretability

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.inspection import permutation_importance
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True, as_frame=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
                                           random_state=42, stratify=y)

# ── Built-in feature importances (tree-based models) ──
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_tr, y_tr)

importances = pd.Series(rf.feature_importances_, index=X.columns)
top10 = importances.nlargest(10).sort_values()

fig, ax = plt.subplots(figsize=(8, 5))
top10.plot.barh(ax=ax, color="steelblue")
ax.set_title("Top 10 Feature Importances (Random Forest)")
ax.set_xlabel("Mean Decrease in Impurity")
plt.tight_layout(); plt.show()

# ── Permutation importance (model-agnostic, less biased) ──
result = permutation_importance(rf, X_te, y_te, n_repeats=10,
                                random_state=42, n_jobs=-1)
perm_imp = pd.Series(result.importances_mean, index=X.columns)
perm_top = perm_imp.nlargest(10).sort_values()

fig, ax = plt.subplots(figsize=(8, 5))
perm_top.plot.barh(ax=ax, color="coral")
ax.set_title("Top 10 Permutation Importances")
ax.set_xlabel("Mean Accuracy Decrease")
plt.tight_layout(); plt.show()

feature_importance.py

Tree-based feature_importances_ (Mean Decrease in Impurity) can be biased towards high-cardinality features. Permutation importance (sklearn.inspection.permutation_importance) is model-agnostic and more reliable — it measures how much the score drops when a feature's values are randomly shuffled. For even deeper explanations use SHAP (pip install shap).

Persisting Pipelines with joblib

import joblib
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("clf",    GradientBoostingClassifier(n_estimators=200, random_state=42)),
])
pipeline.fit(X_tr, y_tr)
print(f"Test accuracy: {pipeline.score(X_te, y_te):.4f}")

# ── Save ──
joblib.dump(pipeline, "breast_cancer_pipeline.joblib")
print("Pipeline saved.")

# ── Load & predict ──
loaded_pipeline = joblib.load("breast_cancer_pipeline.joblib")
predictions  = loaded_pipeline.predict(X_te)
probabilities = loaded_pipeline.predict_proba(X_te)[:, 1]
print(f"Loaded pipeline accuracy: {loaded_pipeline.score(X_te, y_te):.4f}")

persist.py

A joblib file is a serialised Python object — loading a file from an untrusted source can execute arbitrary code. Only load pipelines you created or that come from a trusted, verified source. For production deployments consider the ONNX format for cross-language interoperability and safer serialisation.

Custom Transformers

Extend scikit-learn's pipeline system with your own preprocessing steps by subclassing BaseEstimator and TransformerMixin.

import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

class OutlierClipper(BaseEstimator, TransformerMixin):
    """Clip values outside [mean ± n_std * std] to the boundary."""
    def __init__(self, n_std: float = 3.0):
        self.n_std = n_std

    def fit(self, X, y=None):
        self.mean_ = np.mean(X, axis=0)
        self.std_  = np.std(X, axis=0)
        self.lower_ = self.mean_ - self.n_std * self.std_
        self.upper_ = self.mean_ + self.n_std * self.std_
        return self       # always return self from fit()

    def transform(self, X, y=None):
        return np.clip(X, self.lower_, self.upper_)

class LogTransformer(BaseEstimator, TransformerMixin):
    """Apply log1p to all features (useful for right-skewed data)."""
    def fit(self, X, y=None):
        return self

    def transform(self, X, y=None):
        return np.log1p(np.abs(X))   # handles negatives

# Use in a Pipeline
pipeline = Pipeline([
    ("clipper", OutlierClipper(n_std=2.5)),
    ("log",     LogTransformer()),
    ("scaler",  StandardScaler()),
])

X = np.array([[1, 2], [3, 4], [100, 200], [5, 6]])   # 100,200 are outliers
X_transformed = pipeline.fit_transform(X)
print(X_transformed)

custom_transformer.py

Three rules for custom transformers: (1) fit() computes statistics from training data and stores them as attributes ending in _ (e.g. self.mean_); (2) transform() applies those stored statistics without recomputing; (3) always return self from fit(). This pattern ensures correctness inside Pipeline and cross_val_score.

Best Practices

Always use Pipeline — it prevents data leakage by ensuring preprocessing is fitted only on training data and correctly applied to validation/test data.
Use stratify=y for classification splits — preserves class proportions in both train and test sets, especially critical for imbalanced data.
Prefer cross_val_score over a single train/test split for model evaluation — reduces variance in the estimate by averaging over multiple folds.
Use RandomizedSearchCV first — cover the hyperparameter space broadly, then narrow down with a smaller GridSearchCV around the best region.
Choose metrics that match your problem — accuracy is misleading for imbalanced classes; use F1, ROC-AUC, or precision-recall AUC instead.
Use set_output(transform="pandas") (sklearn ≥ 1.2) — pipelines preserve column names in output DataFrames, making debugging much easier.
Check feature importances and residuals — always inspect what the model learned before deploying; use permutation importance over MDI for reliability.
Version your saved pipelines — include the sklearn version and training date in the filename or metadata; models saved with one sklearn version may not load in another.

Exercises

Exercise 1 — End-to-End Classification Pipeline

Using the Titanic dataset (or any mixed-type CSV), build a complete ML pipeline:

Load data; identify numeric and categorical features.
Build a ColumnTransformer with: median imputation + StandardScaler for numerics; most-frequent imputation + OneHotEncoder for categoricals.
Append a RandomForestClassifier to form the full Pipeline.
Evaluate with 5-fold stratified CV reporting accuracy, F1, and ROC-AUC.
Print a classification_report on the held-out test set.

💡 Hint

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_validate, train_test_split, StratifiedKFold
from sklearn.metrics import classification_report
import pandas as pd

df = pd.read_csv("titanic.csv")[["Survived","Pclass","Sex","Age","SibSp","Parch","Fare","Embarked"]].dropna(subset=["Survived"])
X, y = df.drop("Survived", axis=1), df["Survived"]

num_cols = ["Age", "Fare", "SibSp", "Parch"]
cat_cols = ["Pclass", "Sex", "Embarked"]

pre = ColumnTransformer([
    ("num", Pipeline([("imp", SimpleImputer(strategy="median")), ("sc", StandardScaler())]), num_cols),
    ("cat", Pipeline([("imp", SimpleImputer(strategy="most_frequent")), ("ohe", OneHotEncoder(handle_unknown="ignore"))]), cat_cols),
])
pipe = Pipeline([("pre", pre), ("clf", RandomForestClassifier(n_estimators=100, random_state=42))])

cv = cross_validate(pipe, X, y, cv=StratifiedKFold(5), scoring=["accuracy","f1","roc_auc"])
for k, v in cv.items():
    if k.startswith("test_"):
        print(f"{k}: {v.mean():.3f} ± {v.std():.3f}")

Exercise 2 — Hyperparameter Tuning Showdown

Compare GridSearchCV and RandomizedSearchCV on the same pipeline:

Use the breast cancer dataset and a Pipeline([scaler, GradientBoostingClassifier]).
Define a param_grid for grid search: 3 values each for n_estimators, max_depth, learning_rate (27 combinations).
Define a param_dist for random search: randint and uniform distributions for the same params, n_iter=30.
Time both searches; print best params, best CV ROC-AUC, and wall-clock time.
Which found a better score? Which was faster?

💡 Hint

import time
from scipy.stats import randint, uniform

param_grid = {
    "clf__n_estimators":  [50, 100, 200],
    "clf__max_depth":     [2, 4, 6],
    "clf__learning_rate": [0.05, 0.1, 0.2],
}
param_dist = {
    "clf__n_estimators":  randint(50, 300),
    "clf__max_depth":     randint(2, 8),
    "clf__learning_rate": uniform(0.01, 0.3),
}
for search_cls, params, kwargs in [
    (GridSearchCV,      param_grid, {}),
    (RandomizedSearchCV, param_dist, {"n_iter": 30, "random_state": 42}),
]:
    t0 = time.perf_counter()
    s = search_cls(pipeline, params, cv=5, scoring="roc_auc", n_jobs=-1, **kwargs)
    s.fit(X, y)
    print(f"{search_cls.__name__}: AUC={s.best_score_:.4f}  time={time.perf_counter()-t0:.1f}s")

Exercise 3 — Custom Transformer in a Pipeline

Build a custom transformer and include it in a full pipeline:

Implement TargetEncoder: for each category in a column, replace it with the mean target value of that category (computed on training data only).
It must implement fit(X, y) and transform(X) following the sklearn convention.
Include it in a Pipeline before a Ridge regressor on the California housing dataset.
Compare RMSE with and without target encoding (vs plain OrdinalEncoder).
Verify the transformer works correctly inside cross_val_score — target encoding must be fitted per fold, not on the full dataset.

💡 Hint

import numpy as np
from sklearn.base import BaseEstimator, TransformerMixin

class TargetEncoder(BaseEstimator, TransformerMixin):
    def __init__(self, cols, smoothing=10):
        self.cols = cols
        self.smoothing = smoothing

    def fit(self, X, y):
        import pandas as pd
        df = pd.DataFrame(X, columns=range(X.shape[1]))
        self.global_mean_ = np.mean(y)
        self.encodings_ = {}
        for col in self.cols:
            stats = pd.DataFrame({"y": y}).groupby(df[col])["y"].agg(["mean","count"])
            smooth = (stats["count"] * stats["mean"] + self.smoothing * self.global_mean_) / (stats["count"] + self.smoothing)
            self.encodings_[col] = smooth.to_dict()
        return self

    def transform(self, X, y=None):
        import pandas as pd
        df = pd.DataFrame(X, columns=range(X.shape[1])).copy()
        for col in self.cols:
            df[col] = df[col].map(self.encodings_[col]).fillna(self.global_mean_)
        return df.values.astype(float)