🎯 What You'll Learn

  • Understand why ensembles work through the lens of bias-variance decomposition and the wisdom of crowds
  • Build voting ensembles (hard and soft) with VotingClassifier
  • Explain bootstrap sampling and implement BaggingClassifier with out-of-bag evaluation
  • Train RandomForestClassifier and understand why random feature subsets make trees more diverse
  • Extract and interpret feature importances from a Random Forest
  • Combine diverse model types with StackingClassifier, letting a trained meta-learner replace a fixed voting rule

1 Why Ensembles Work: Wisdom of Crowds

In 1906, statistician Francis Galton attended a county fair where 800 people guessed the weight of an ox. No single guess was exact, but the median of all guesses — 1207 pounds — was accurate to within 0.8% of the true weight of 1198 pounds. Individual judgment was imperfect; collective judgment was astonishing.

Machine learning ensembles work on the same principle: individual models make different mistakes, and those errors tend to cancel out when you aggregate predictions. This only works if the models are:

  • Diverse — they make different mistakes on different examples
  • Better than random — each individual model has some predictive power
In [1]:
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier

np.random.seed(42)
X, y = make_classification(
    n_samples=2000, n_features=20, n_informative=10,
    n_redundant=5, random_state=42)

# Individual model cross-validation scores
models = {
    'Logistic Regression': LogisticRegression(max_iter=1000, random_state=42),
    'Decision Tree':       DecisionTreeClassifier(max_depth=5, random_state=42),
    'KNN (k=7)':           KNeighborsClassifier(n_neighbors=7),
}

print("Individual model cross-validation accuracy:")
for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=5, scoring='roc_auc')
    print(f"  {name:25s}: {scores.mean():.4f} ± {scores.std():.4f}")

# Logistic Regression     : 0.8712 ± 0.0183
# Decision Tree           : 0.8419 ± 0.0215
# KNN (k=7)               : 0.8534 ± 0.0201
# None of these are the best model — but combining them often beats all three
🔑
Bias-Variance Decomposition of Ensemble Error

The expected prediction error of any model decomposes as: Error = Bias² + Variance + Irreducible Noise. A single deep decision tree has low bias but high variance (it changes a lot if you retrain on different data). Averaging many such trees keeps the low bias while canceling out their variance — the trees' individual errors wash out. Ensembles primarily reduce variance.

2 Majority Voting (Hard Voting)

The simplest ensemble: train multiple different models and predict whichever class the majority of models predict. Three models vote; the class that gets two or more votes wins. Scikit-learn's VotingClassifier handles this automatically.

In [2]:
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split, cross_val_score
import numpy as np

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

# Note: KNN and Logistic Regression need scaled features
lr_pipe  = Pipeline([('scaler', StandardScaler()), ('lr', LogisticRegression(max_iter=1000, random_state=42))])
knn_pipe = Pipeline([('scaler', StandardScaler()), ('knn', KNeighborsClassifier(n_neighbors=7))])
dt       = DecisionTreeClassifier(max_depth=5, random_state=42)

# Hard voting: each model casts one vote; majority wins
hard_voter = VotingClassifier(
    estimators=[
        ('lr',  lr_pipe),
        ('dt',  dt),
        ('knn', knn_pipe),
    ],
    voting='hard'
)
hard_voter.fit(X_train, y_train)

print(f"Hard Voting  — Test accuracy: {hard_voter.score(X_test, y_test):.4f}")
from sklearn.metrics import roc_auc_score
y_pred_h = hard_voter.predict(X_test)
# Note: hard voting doesn't support predict_proba for AUC directly

# Compare to individual models
for name, model in [('Logistic Regression', lr_pipe), ('Decision Tree', dt), ('KNN', knn_pipe)]:
    model.fit(X_train, y_train)
    print(f"  {name:25s}: {model.score(X_test, y_test):.4f}")

# Hard Voting         : 0.8900  ← usually beats all individuals
# Logistic Regression : 0.8725
# Decision Tree       : 0.8450
# KNN                 : 0.8600

3 Soft Voting

Soft voting averages the predicted probabilities across all models and picks the class with the highest average probability. Because it uses full confidence information rather than just the winning class, soft voting almost always outperforms hard voting — provided each model is well-calibrated.

In [3]:
from sklearn.ensemble import VotingClassifier
from sklearn.metrics import roc_auc_score
import numpy as np

# Soft voting: average predict_proba across models
soft_voter = VotingClassifier(
    estimators=[
        ('lr',  lr_pipe),
        ('dt',  dt),
        ('knn', knn_pipe),
    ],
    voting='soft'
)
soft_voter.fit(X_train, y_train)

proba_soft = soft_voter.predict_proba(X_test)[:, 1]
auc_soft   = roc_auc_score(y_test, proba_soft)
acc_soft   = soft_voter.score(X_test, y_test)

print(f"Soft Voting — Test accuracy: {acc_soft:.4f}  AUC: {auc_soft:.4f}")

# Inspect the averaged probabilities for 5 test samples
for i in range(5):
    lr_p  = lr_pipe.predict_proba(X_test[i:i+1])[0, 1]
    dt_p  = dt.predict_proba(X_test[i:i+1])[0, 1]
    knn_p = knn_pipe.predict_proba(X_test[i:i+1])[0, 1]
    avg_p = (lr_p + dt_p + knn_p) / 3
    truth = y_test[i]
    print(f"  Sample {i}: LR={lr_p:.3f}  DT={dt_p:.3f}  KNN={knn_p:.3f}  avg={avg_p:.3f}  true={truth}")

# Soft Voting — Test accuracy: 0.9000  AUC: 0.9402
# Higher AUC than hard voting because it uses full probability information

The table below mirrors the five test samples printed above: each base model's predicted probability of the positive class, and the averaged probability that soft voting actually uses to make its decision. Pick a sample to see how three individually-uncertain models combine into one confident ensemble call:

Each bar is one model's P(class=1); the dashed line is the soft-voting average that determines the final prediction.

💡
When to Use Hard vs Soft Voting

Use soft voting whenever all your base estimators support predict_proba. It's almost always better because it leverages the confidence of each model's prediction. Use hard voting only when some of your base estimators don't expose probabilities (e.g., some SVM variants). For AUC-focused tasks, soft voting is mandatory since AUC requires predicted probabilities.

4 Weighted Voting

Not all models are equally good. A well-calibrated approach is to give more influence to the models that performed better on a held-out validation set. This is weighted voting.

In [4]:
from sklearn.ensemble import VotingClassifier
from sklearn.model_selection import cross_val_score
import numpy as np

# First evaluate each model's AUC on the training data via cross-validation
# to determine appropriate weights
base_aucs = {}
for name, model in [('lr', lr_pipe), ('dt', dt), ('knn', knn_pipe)]:
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring='roc_auc')
    base_aucs[name] = scores.mean()
    print(f"  {name}: CV-AUC = {scores.mean():.4f}")

# Weights proportional to CV-AUC
weights = [base_aucs['lr'], base_aucs['dt'], base_aucs['knn']]
print(f"\nWeights: {[round(w, 3) for w in weights]}")

weighted_voter = VotingClassifier(
    estimators=[('lr', lr_pipe), ('dt', dt), ('knn', knn_pipe)],
    voting='soft',
    weights=weights   # higher-AUC models get more say
)
weighted_voter.fit(X_train, y_train)

from sklearn.metrics import roc_auc_score
proba_w = weighted_voter.predict_proba(X_test)[:, 1]
print(f"\nWeighted Soft Voting — AUC: {roc_auc_score(y_test, proba_w):.4f}")
# Typically same or slightly better than unweighted soft voting

5 Bagging: Bootstrap Aggregating

Voting combines different model types. Bagging takes a different approach: train the same algorithm multiple times on different bootstrap samples of the training data — random samples drawn with replacement. Because each tree sees a slightly different dataset, the trees make different errors that cancel out when averaged.

Bootstrapping: from n training samples, draw n samples with replacement. On average, each bootstrap sample contains ~63.2% of original samples (some appear multiple times). The ~36.8% that weren't drawn become the out-of-bag (OOB) samples for that tree — a free validation set.

Original training set: n samples Original Dataset (n samples) resample Bootstrap sample 1: n draws with replacement from the original dataset Bootstrap Set 1 ~63% unique rows Bootstrap sample 2: a different random resample Bootstrap Set 2 ~63% unique rows Bootstrap sample 3: a different random resample Bootstrap Set 3 ~63% unique rows Bootstrap sample N: a different random resample Bootstrap Set N ~63% unique rows train Tree 1 trained only on Bootstrap Set 1 Tree 1 trained in parallel Tree 2 trained only on Bootstrap Set 2 Tree 2 trained in parallel Tree 3 trained only on Bootstrap Set 3 Tree 3 trained in parallel Tree N trained only on Bootstrap Set N Tree N trained in parallel Combine all predictions: majority vote for classification, average for regression Combine Majority Vote / Average e.g. n = 1000 rows

Bagging end-to-end: the original dataset is resampled with replacement into N bootstrap subsets (each ~63% unique rows), one model is trained per subset entirely independently and in parallel, and their predictions are combined — majority vote for classification, simple average for regression. This is the core mental model behind every algorithm in this lesson, including Random Forests.

In [5]:
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import roc_auc_score
import numpy as np

# Bagging: 100 decision trees on different bootstrap samples
bag = BaggingClassifier(
    estimator=DecisionTreeClassifier(max_depth=None),  # deep trees: high variance
    n_estimators=100,
    max_samples=1.0,    # each bootstrap uses 100% of n (with replacement)
    max_features=1.0,   # all features (feature randomness is added in RandomForest)
    bootstrap=True,
    oob_score=True,     # use OOB samples for free validation estimate
    random_state=42,
    n_jobs=-1
)
bag.fit(X_train, y_train)

print(f"Bagging — Test accuracy:   {bag.score(X_test, y_test):.4f}")
print(f"Bagging — OOB accuracy:    {bag.oob_score_:.4f}")  # no test set needed!
proba_bag = bag.predict_proba(X_test)[:, 1]
print(f"Bagging — Test AUC:        {roc_auc_score(y_test, proba_bag):.4f}")

# Compare to single deep tree (high variance)
single_deep = DecisionTreeClassifier(max_depth=None, random_state=42)
single_deep.fit(X_train, y_train)
print(f"\nSingle deep tree — Test accuracy: {single_deep.score(X_test, y_test):.4f}")
print(f"Single deep tree — Train accuracy: {single_deep.score(X_train, y_train):.4f}")

# Bagging — Test accuracy:   0.9050     ← 9% better than single tree
# Bagging — OOB accuracy:    0.8980     ← close to test (OOB is reliable)
# Single deep tree — Test:   0.8250
# Single deep tree — Train:  1.0000     ← perfect train, poor test = high variance
🔑
OOB Score: Free Cross-Validation

Each tree in a Bagging ensemble is trained on ~63% of the data. The remaining ~37% — the out-of-bag samples — can evaluate each tree. Averaging across all trees gives an unbiased performance estimate without a separate held-out set. With large ensembles (≥100 trees), OOB score is a reliable alternative to 5-fold cross-validation and comes for free.

6 Random Forests

Random Forests extend Bagging with one crucial addition: at each split, instead of considering all features, each tree considers only a random subset of features. This injects an extra layer of diversity — even trees trained on similar bootstrap samples will use different features at each node.

Why does this help? Without feature randomness, if one feature is very dominant, all trees will use it at the root and look very similar. With feature randomness, other features get more opportunities to shine, and the trees become less correlated. Less correlated trees → averaging reduces more variance.

In [6]:
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, roc_auc_score
import numpy as np

# Random Forest: bagging + random feature subset at each split
rf = RandomForestClassifier(
    n_estimators=200,      # more trees = better (diminishing returns after ~200)
    max_features='sqrt',   # consider sqrt(n_features) features per split — classification default
    max_depth=None,        # deep trees: RF relies on diversity, not shallow trees
    min_samples_leaf=1,
    bootstrap=True,
    oob_score=True,
    n_jobs=-1,
    random_state=42
)
rf.fit(X_train, y_train)

print(f"Random Forest — Test accuracy: {rf.score(X_test, y_test):.4f}")
print(f"Random Forest — OOB accuracy:  {rf.oob_score_:.4f}")
proba_rf = rf.predict_proba(X_test)[:, 1]
print(f"Random Forest — Test AUC:      {roc_auc_score(y_test, proba_rf):.4f}")
print(f"\n{classification_report(y_test, rf.predict(X_test))}")

# Effect of n_estimators
print("\nEffect of n_estimators on test AUC:")
for n_est in [10, 25, 50, 100, 200, 500]:
    rf_n = RandomForestClassifier(n_estimators=n_est, max_features='sqrt',
                                   oob_score=True, n_jobs=-1, random_state=42)
    rf_n.fit(X_train, y_train)
    auc = roc_auc_score(y_test, rf_n.predict_proba(X_test)[:, 1])
    print(f"  n_estimators={n_est:4d}: AUC={auc:.4f}  OOB={rf_n.oob_score_:.4f}")

# n_estimators=  10: AUC=0.9301  OOB=0.9012  ← high variance
# n_estimators= 100: AUC=0.9489  OOB=0.9223
# n_estimators= 200: AUC=0.9501  OOB=0.9241  ← good balance
# n_estimators= 500: AUC=0.9507  OOB=0.9249  ← diminishing returns

7 Key Random Forest Hyperparameters

In [7]:
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
import numpy as np

# Tuning max_features
print("Effect of max_features:")
for mf in [1, 3, 'sqrt', 'log2', None]:  # None = all features (= Bagging)
    rf = RandomForestClassifier(n_estimators=100, max_features=mf, n_jobs=-1, random_state=42)
    auc = cross_val_score(rf, X, y, cv=5, scoring='roc_auc').mean()
    print(f"  max_features={str(mf):8s}: CV-AUC = {auc:.4f}")

# max_features=1      : 0.9050  ← too restrictive, each tree is weak
# max_features=sqrt   : 0.9421  ← classification default, usually best
# max_features=log2   : 0.9398
# max_features=None   : 0.9312  ← all features = correlated trees (plain bagging)

# Tuning max_depth
print("\nEffect of max_depth:")
for md in [3, 5, 10, 20, None]:
    rf = RandomForestClassifier(n_estimators=100, max_depth=md, max_features='sqrt',
                                 n_jobs=-1, random_state=42)
    scores = cross_val_score(rf, X, y, cv=5, scoring='roc_auc')
    print(f"  max_depth={str(md):6s}: CV-AUC = {scores.mean():.4f} ± {scores.std():.4f}")

# max_depth=3   : 0.8941 ± 0.0189  ← underfitting, too shallow
# max_depth=10  : 0.9352 ± 0.0143
# max_depth=None: 0.9421 ± 0.0162  ← deep trees work well with RF diversity
Hyperparameter Default Tuning guidance
n_estimators 100 More is better; use OOB to find diminishing returns point
max_features 'sqrt' 'sqrt' for classification; n_features/3 for regression
max_depth None Usually keep None; shallow trees can reduce RF performance
min_samples_leaf 1 Try 1–20; higher → smoother predictions, less overfit
oob_score False Set True to get free OOB performance estimate

8 Feature Importance in Random Forests

Random Forest feature importances are averages of the single-tree importances across all 100+ trees. Because different trees use different bootstrap samples and different feature subsets, the RF importance is generally more stable and reliable than a single tree's importance — though still biased toward high-cardinality features.

In [8]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

bc = load_breast_cancer()
X_bc, y_bc = bc.data, bc.target
feature_names = bc.feature_names
X_btr, X_bte, y_btr, y_bte = train_test_split(X_bc, y_bc, test_size=0.2, random_state=42)

rf_bc = RandomForestClassifier(n_estimators=200, max_features='sqrt',
                                 oob_score=True, n_jobs=-1, random_state=42)
rf_bc.fit(X_btr, y_btr)

# Impurity-based feature importance (fast, but can be biased)
imp_df = pd.Series(rf_bc.feature_importances_, index=feature_names)
print("Top 10 RF feature importances (impurity-based):")
print(imp_df.nlargest(10).round(4))

# Permutation importance (slower, but unbiased)
result = permutation_importance(rf_bc, X_bte, y_bte, n_repeats=20,
                                 random_state=42, n_jobs=-1)
perm_df = pd.Series(result.importances_mean, index=feature_names)
print("\nTop 10 permutation importances (on test set):")
print(perm_df.nlargest(10).round(4))

# Compare: do they agree?
top_imp  = set(imp_df.nlargest(10).index)
top_perm = set(perm_df.nlargest(10).index)
print(f"\nFeatures in both top-10 lists: {len(top_imp & top_perm)}/10")
# Usually 7-9 overlap — permutation importance is the more trustworthy version

The fraud-detection model later in this lesson prints exactly this kind of ranking. Here are its top 5 fraud predictors as a horizontal bar chart — the standard way to visualize feature_importances_ (sorted, longest bar on top):

Random Forest impurity-based feature importances for the fraud-detection model (Real-World Spotlight, below). Importances sum to 1.0 across all features; only the top 5 are shown.

⚠️
Use Permutation Importance for Critical Feature Selection

Impurity-based feature importances are computed on training data — they reflect how useful a feature was during training, not how much it matters for test-set predictions. Permutation importance shuffles one feature at a time on the test set and measures performance degradation. It is unbiased and reflects true generalization importance. The extra computation time is almost always worth it.

9 Stacking: Learning How to Combine Models

Voting (Sections 2–4) combines predictions with a fixed rule — majority vote, or a hand-picked weighted average. Stacking (stacked generalization) replaces that fixed rule with a learned one: train a second model, the meta-learner, whose job is to figure out the best way to combine the base models' predictions, using the base models' outputs as its input features.

In [9]:
import numpy as np
from sklearn.ensemble import StackingClassifier, RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=1000, n_features=20, n_informative=10, random_state=42)

base_learners = [
    ('rf',  RandomForestClassifier(n_estimators=100, random_state=42)),
    ('gb',  GradientBoostingClassifier(n_estimators=100, random_state=42)),
    ('svc', SVC(probability=True, random_state=42)),
]

# The meta-learner (final_estimator) is trained on the base learners' OUT-OF-FOLD
# predictions -- sklearn handles this internally via cross-validation (cv=5) so
# the meta-learner never sees predictions a base model made on its own training data.
stack = StackingClassifier(
    estimators=base_learners,
    final_estimator=LogisticRegression(),
    cv=5,
)

# Compare each base learner alone vs the stacked ensemble
for name, model in base_learners:
    scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
    print(f"{name:5s} alone:  {scores.mean():.4f} ± {scores.std():.4f}")

stack_scores = cross_val_score(stack, X, y, cv=5, scoring='accuracy')
print(f"\nStacked ensemble: {stack_scores.mean():.4f} ± {stack_scores.std():.4f}")
# Stacking typically beats every individual base learner, and often beats
# simple voting too -- the meta-learner can weigh SVC more heavily on
# examples where trees struggle, and vice versa, instead of a fixed blend.
🔑
Why Out-of-Fold Predictions Matter

If the meta-learner trained on predictions the base models made on their own training data, it would be learning from overfit, overly-confident predictions that don't reflect real generalization error — and the whole ensemble would overfit. StackingClassifier's cv parameter fixes this automatically: each base model's contribution to the meta-learner's training set comes from when that example was in a held-out fold, exactly like cross-validation's train/test discipline (Lesson 20), just nested one level deeper.

Stacking's practical cost is complexity and inference latency — you're now running every base model plus a meta-learner for every prediction — so it earns its place when the accuracy gain over the best single model or a simple voting ensemble is worth that overhead: Kaggle-style competitions where every fraction of a point matters, or high-value predictions (credit risk, fraud) where marginal accuracy translates directly into money.

Ensembling approach Combination rule When it shines
Voting (Sections 2–4) Fixed — majority or weighted average Simple, fast, hard to overfit; good default
Bagging / Random Forest (Sections 5–8) Average over many same-type models on bootstrapped data Reducing variance of a single high-variance model type
Stacking Learned — a trained meta-model Squeezing out the last bit of accuracy from diverse model types
🌍

Real-World Spotlight: Fraud Detection with Random Forest

Online fraud datasets are characterized by severe class imbalance (often 0.1–1% fraud), and both false positives (blocking legitimate customers) and false negatives (missing fraud) are costly. Random Forests with class_weight='balanced' and OOB evaluation are a standard strong baseline in the industry.

In [10]:
import numpy as np
import pandas as pd
from sklearn.ensemble import VotingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
from sklearn.metrics import roc_auc_score, average_precision_score, classification_report

np.random.seed(42)
n = 100_000
n_fraud = int(n * 0.005)  # 0.5% fraud rate

# Synthetic transaction data
X_legit = np.random.randn(n - n_fraud, 20) * 0.8
X_fraud = np.random.randn(n_fraud, 20) * 1.5 + 0.3   # fraud skewed in feature space
X_fraud = np.random.randn(n, 20)
# Add informative features for fraud signal
for i in range(5):
    X_fraud[: , i] += np.random.choice([0, 1], n) * 1.2

y = np.zeros(n)
fraud_idx = np.random.choice(n, n_fraud, replace=False)
y[fraud_idx] = 1
feature_names = ['transaction_amount', 'merchant_category', 'time_since_last_txn',
                  'dist_from_home', 'device_age'] + [f'feat_{i}' for i in range(15)]

X_frtr, X_frte, y_frtr, y_frte = train_test_split(
    X_fraud, y, test_size=0.2, stratify=y, random_state=42)
print(f"Train fraud rate: {y_frtr.mean():.3%}  |  Test fraud rate: {y_frte.mean():.3%}")

# Random Forest with balanced class weights
rf_fraud = RandomForestClassifier(
    n_estimators=200,
    max_features='sqrt',
    class_weight='balanced',    # compensates for imbalance
    oob_score=True,
    n_jobs=-1,
    random_state=42
)
rf_fraud.fit(X_frtr, y_frtr)

proba_rf = rf_fraud.predict_proba(X_frte)[:, 1]
print(f"\nRandom Forest:")
print(f"  OOB accuracy:    {rf_fraud.oob_score_:.4f}")
print(f"  Test AUC-ROC:    {roc_auc_score(y_frte, proba_rf):.4f}")
print(f"  Avg Precision:   {average_precision_score(y_frte, proba_rf):.4f}")

# Top feature importances
imp = pd.Series(rf_fraud.feature_importances_, index=feature_names)
print("\nTop 5 fraud predictors:")
print(imp.nlargest(5).round(4))
# transaction_amount   0.1823
# time_since_last_txn  0.1619
# merchant_category    0.1401

# Voting ensemble: LR + RF + compare
lr_pipe = Pipeline([('sc', StandardScaler()), ('lr', LogisticRegression(class_weight='balanced', max_iter=1000))])
voter = VotingClassifier(
    estimators=[('lr', lr_pipe), ('rf', rf_fraud)],
    voting='soft', weights=[1, 2]   # RF gets double vote
)
voter.fit(X_frtr, y_frtr)
proba_v = voter.predict_proba(X_frte)[:, 1]
print(f"\nVoting Ensemble (LR + RF, weight=[1,2]):")
print(f"  Test AUC-ROC: {roc_auc_score(y_frte, proba_v):.4f}")

The Random Forest with balanced class weights provides a strong, interpretable baseline: transaction amount, time since last transaction, and merchant category are the top fraud predictors — exactly what domain experts would expect. The voting ensemble with a logistic regression component adds modest AUC improvement by blending the RF's nonlinear patterns with LR's globally calibrated probabilities.

✍️ Practice Exercises

  1. Build a VotingClassifier using Logistic Regression, Decision Tree (depth=5), and Naive Bayes on the Breast Cancer dataset. Compare hard vs soft voting AUC. Which performs better and why?
  2. Train a BaggingClassifier with oob_score=True on the Wine dataset. Check that OOB accuracy is close to 5-fold cross-validation accuracy.
  3. For a Random Forest on the Titanic dataset, plot the test AUC vs n_estimators for values [5, 10, 25, 50, 100, 200]. Where does performance plateau?
  4. Compare feature_importances_ and permutation_importance on the Diabetes dataset. Do they give different rankings for the top features?

📚 Primary Source for This Lesson

scikit-learn: Ensemble Methods
The comprehensive guide covering Voting, Bagging, Random Forests, and beyond. For the theoretical foundations: Breiman (2001) "Random Forests" (Machine Learning 45:5–32) — the original paper is surprisingly readable and worth 30 minutes of your time. Also see Dietterich (2000) "Ensemble Methods in Machine Learning" for a broader theoretical overview.

💬 Confused about why OOB score differs from cross-validation? Getting worse results with the ensemble than individual models? Paste your situation and your tutor will diagnose why your models may not be diverse enough.