Boosting: AdaBoost, Gradient Boosting, XGBoost & LightGBM — ML & DL Mastery

🎯 What You'll Learn

Contrast Boosting with Bagging: sequential vs parallel, bias vs variance reduction
Understand how AdaBoost reweights misclassified samples to focus on hard examples
Grasp Gradient Boosting as gradient descent in function space — fitting residuals iteratively
Use GradientBoostingClassifier, XGBClassifier, and LGBMClassifier with key hyperparameters
Apply early stopping and learning rate scheduling to prevent overfitting

1 Boosting vs Bagging

Bagging and Boosting are both ensemble strategies, but they attack the bias-variance problem from opposite angles:

Property	Bagging (Random Forest)	Boosting (GBM, XGBoost)
Training order	Parallel — all trees trained independently	Sequential — each tree corrects the previous
Base learner	Deep trees (high variance, low bias)	Shallow trees, depth 3–6 (weak learners)
What it reduces	Variance — averages out noise	Bias — each tree corrects remaining errors
Overfit risk	Low — averaging is self-regularizing	Moderate — can overfit if too many trees or deep
Typical accuracy	Very good with little tuning	State-of-the-art on tabular data with tuning

🔑

Boosting = Ensemble of Weak Learners that Become Strong

A "weak learner" is any model only slightly better than random guessing. The magic of boosting is that combining many such weak learners — where each one corrects the errors of the combined model so far — can produce a very accurate "strong" ensemble. The theoretical guarantee (Schapire, 1990) is that if you can generate weak learners consistently better than random, you can boost them to arbitrary accuracy.

The clearest way to see the difference is to look at the two architectures side by side. Bagging fans the data out to independent trees that never see each other's output; Boosting threads the data through a chain where every link is trained to fix what the previous link got wrong.

Bagging (top) trains every tree independently on a bootstrap sample and averages the results — parallel, variance-reducing. Boosting (bottom) trains each weak learner h_t sequentially on the residuals left by F_t-1, then adds a shrunk copy (η·h_t) into a running weighted sum — sequential, bias-reducing. Tree 3 in boosting cannot start until Tree 2's mistakes are known.

2 AdaBoost: Adaptive Boosting

AdaBoost (Adaptive Boosting) was the original boosting algorithm (Freund & Schapire, 1997). The algorithm:

Assign equal sample weights w_i = 1/n to all training samples
Train a weak learner on the weighted samples
Compute the weighted error rate; calculate the learner's "say" (α) based on its accuracy
Increase weights of misclassified samples (so the next learner focuses on them)
Decrease weights of correctly classified samples
Repeat steps 2–5; combine all learners with weighted vote: F(x) = Σ αₜ · hₜ(x)

In [1]:

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import roc_auc_score
import numpy as np

np.random.seed(42)
X, y = make_classification(
    n_samples=2000, n_features=20, n_informative=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

# Default base estimator: DecisionTreeClassifier(max_depth=1) — a "stump"
ada = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1),  # decision stump
    n_estimators=200,
    learning_rate=1.0,     # shrinks each tree's contribution
    algorithm='SAMME.R',   # use probability estimates (better than SAMME)
    random_state=42
)
ada.fit(X_train, y_train)

print(f"AdaBoost — Test accuracy: {ada.score(X_test, y_test):.4f}")
print(f"AdaBoost — Test AUC:      {roc_auc_score(y_test, ada.predict_proba(X_test)[:,1]):.4f}")

# Staged evaluation: how does performance improve with more estimators?
print("\nStagewise performance:")
staged_auc = []
for i, proba in enumerate(ada.staged_predict_proba(X_test)):
    auc = roc_auc_score(y_test, proba[:, 1])
    staged_auc.append(auc)

import matplotlib.pyplot as plt
plt.figure(figsize=(10, 4))
plt.plot(range(1, 201), staged_auc, color='steelblue')
plt.xlabel('Number of estimators')
plt.ylabel('Test AUC')
plt.title('AdaBoost: Performance vs Number of Estimators')
plt.axhline(y=max(staged_auc), color='red', linestyle='--', alpha=0.6,
            label=f'Max AUC={max(staged_auc):.4f} at n={staged_auc.index(max(staged_auc))+1}')
plt.legend()
plt.tight_layout()
plt.savefig('adaboost_staged.png', dpi=150, bbox_inches='tight')
print(f"Best AUC: {max(staged_auc):.4f} at n_estimators={staged_auc.index(max(staged_auc))+1}")

3 Gradient Boosting: The Core Idea

Gradient Boosting (Friedman, 2001) reframes boosting as gradient descent in function space. Instead of reweighting samples, each new tree is fit to the residuals — the errors of the current ensemble.

Step by step:

Initialize with a simple prediction F₀(x) = mean(y)
Compute residuals: rᵢ = yᵢ − F₀(xᵢ)
Fit a tree h₁(x) to predict the residuals
Update: F₁(x) = F₀(x) + η·h₁(x), where η is the learning rate
Compute new residuals on F₁; fit h₂; update to F₂; repeat T times
Final prediction: F(x) = F₀(x) + η·Σ hₜ(x)

In [2]:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeRegressor

# Illustrate residual fitting manually on a regression problem
np.random.seed(42)
X_1d = np.sort(np.random.uniform(0, 5, 100)).reshape(-1, 1)
y_1d = np.sin(X_1d.ravel()) + np.random.randn(100) * 0.2

# Stage 0: initialize with mean
F0 = np.full(len(y_1d), y_1d.mean())
lr = 0.5

preds = F0.copy()
for stage in range(1, 6):  # 5 trees
    residuals = y_1d - preds
    # Fit a shallow tree to the residuals
    tree = DecisionTreeRegressor(max_depth=2)
    tree.fit(X_1d, residuals)
    # Update predictions
    preds = preds + lr * tree.predict(X_1d)
    mse = np.mean((y_1d - preds)**2)
    print(f"After tree {stage}: MSE = {mse:.4f}  (residual std = {residuals.std():.4f})")

# After tree 1: MSE = 0.1823  (residual std = 0.5234)
# After tree 2: MSE = 0.1231  (residual std = 0.3894)
# After tree 3: MSE = 0.0872  (residual std = 0.3024)
# After tree 4: MSE = 0.0641  (residual std = 0.2507)
# After tree 5: MSE = 0.0491  (residual std = 0.2163)
# Each tree corrects the errors of all previous trees

💡

Gradient Boosting Generalises Residual Fitting

For regression with MSE loss, the gradient is exactly the residual, so "fitting residuals" is the same as gradient descent. For other losses (log-loss, MAE, Huber), you fit the gradient of that loss function — which may not exactly equal the residuals. This is why gradient boosting generalises to any differentiable loss function, making it applicable to regression, classification, ranking, and custom business metrics.

Watch It Happen: Boosting a 1D Regression, Round by Round

Below is gradient boosting running live in your browser on 18 noisy points sampled from a non-linear function. Each "weak learner" is a decision stump — a single split found by brute-force search over candidate thresholds, predicting a constant value on each side. Drag the slider to add more boosting rounds and watch the cumulative model F_t(x) = F₀(x) + η·Σh_t(x) bend to fit the data, while the residual panel shows the errors it still has left to correct shrinking round by round:

Boosting rounds (T) 1

Round 1 — a single stump roughly splits the data into a low half and a high half.

Residuals after round 1 — still large; most of the signal hasn't been captured yet.

Training MSE vs boosting round — note the steep early drop and diminishing returns as more rounds are added (the shape that motivates early stopping).

4 Gradient Boosting in scikit-learn

In [3]:

from sklearn.ensemble import GradientBoostingClassifier, GradientBoostingRegressor
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import roc_auc_score, classification_report
import numpy as np

X_bc, y_bc = load_breast_cancer(return_X_y=True)
X_btr, X_bte, y_btr, y_bte = train_test_split(X_bc, y_bc, test_size=0.2, random_state=42)

# Key hyperparameters:
# n_estimators:  number of trees (more = lower bias, more overfit risk)
# learning_rate: shrinks each tree's contribution (smaller → more trees needed)
# max_depth:     complexity of each tree (3–5 typical)
# subsample:     fraction of samples per tree (< 1.0 = stochastic GBM)
# min_samples_leaf: prevents overfitting in individual trees

gbm = GradientBoostingClassifier(
    n_estimators=200,
    learning_rate=0.1,
    max_depth=3,
    subsample=0.8,         # stochastic GB: use 80% of data per tree
    min_samples_leaf=10,
    random_state=42
)
gbm.fit(X_btr, y_btr)

proba_gbm = gbm.predict_proba(X_bte)[:, 1]
print(f"GBM — Test accuracy: {gbm.score(X_bte, y_bte):.4f}")
print(f"GBM — Test AUC:      {roc_auc_score(y_bte, proba_gbm):.4f}")

# Compare learning_rate vs n_estimators tradeoff
print("\nLearning rate vs n_estimators (same total 'capacity'):")
configs = [
    (0.5, 40), (0.2, 100), (0.1, 200), (0.05, 400), (0.01, 2000)
]
for lr, n in configs:
    if n <= 500:  # skip long ones in demo
        g = GradientBoostingClassifier(learning_rate=lr, n_estimators=n,
                                        max_depth=3, random_state=42)
        auc = cross_val_score(g, X_bc, y_bc, cv=5, scoring='roc_auc').mean()
        print(f"  lr={lr:.2f}, n_est={n:4d}: CV-AUC = {auc:.4f}")

# lr=0.50, n_est=  40: CV-AUC = 0.9721
# lr=0.20, n_est= 100: CV-AUC = 0.9804
# lr=0.10, n_est= 200: CV-AUC = 0.9843  ← sweet spot
# lr=0.05, n_est= 400: CV-AUC = 0.9851  ← marginal improvement, 2× slower

5 XGBoost: Optimized Gradient Boosting

XGBoost (Chen & Guestrin, 2016) won dozens of Kaggle competitions and revolutionized tabular ML. It extends gradient boosting with:

Regularization: L1 (reg_alpha) and L2 (reg_lambda) penalties on leaf weights
Second-order gradients: uses both gradient and Hessian for more precise updates
Native missing value handling: learns the optimal direction for NaN values
Column and row subsampling: stochastic boosting similar to Random Forest
Early stopping: stops when validation performance stops improving

In [4]:

import xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from sklearn.datasets import load_breast_cancer
import numpy as np

X_bc, y_bc = load_breast_cancer(return_X_y=True)
X_btr, X_bte, y_btr, y_bte = train_test_split(X_bc, y_bc, test_size=0.2, random_state=42)
# Further split training into train + validation for early stopping
X_btr2, X_bval, y_btr2, y_bval = train_test_split(
    X_btr, y_btr, test_size=0.15, random_state=42)

# XGBoost classifier
xgb_cls = xgb.XGBClassifier(
    n_estimators=1000,          # large; early stopping will find the right point
    learning_rate=0.05,
    max_depth=4,
    subsample=0.8,              # 80% of rows per tree
    colsample_bytree=0.8,       # 80% of features per tree
    reg_alpha=0.1,              # L1 regularization on leaf weights
    reg_lambda=1.0,             # L2 regularization on leaf weights
    eval_metric='auc',
    early_stopping_rounds=50,   # stop if no improvement for 50 rounds
    n_jobs=-1,
    random_state=42
)
xgb_cls.fit(
    X_btr2, y_btr2,
    eval_set=[(X_bval, y_bval)],
    verbose=50   # print every 50 rounds
)

print(f"\nBest iteration:       {xgb_cls.best_iteration}")
print(f"Best validation AUC:  {xgb_cls.best_score:.4f}")
proba_xgb = xgb_cls.predict_proba(X_bte)[:, 1]
print(f"Final test AUC:       {roc_auc_score(y_bte, proba_xgb):.4f}")

# XGBoost handles missing values natively
import pandas as pd
X_with_nan = X_bte.copy()
X_with_nan[np.random.rand(*X_with_nan.shape) < 0.1] = np.nan  # 10% missing
proba_nan = xgb_cls.predict_proba(X_with_nan)[:, 1]
print(f"Test AUC with 10% NaN: {roc_auc_score(y_bte, proba_nan):.4f}")
# AUC barely changes — XGBoost handles NaN natively!

⚠️

Always Use Early Stopping with XGBoost

Set a large n_estimators (500–2000) and let early stopping find the optimal number. Training without early stopping and manually choosing n_estimators via grid search is much slower and less reliable. The pattern: large n_estimators + early_stopping_rounds=50 + eval_set=[(X_val, y_val)]. The final model uses best_iteration trees automatically.

6 LightGBM: Leaf-Wise Growth

LightGBM (Ke et al., 2017) introduced leaf-wise tree growth. Scikit-learn and XGBoost grow trees level-wise: all leaves at depth d are split before moving to depth d+1. LightGBM grows leaf-wise: always split the single leaf with the largest gain, regardless of depth. This allows it to achieve lower loss with fewer splits — it focuses resources exactly where they help most.

In [5]:

import lightgbm as lgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
import numpy as np

X_btr3, X_bval3, y_btr3, y_bval3 = train_test_split(
    X_btr, y_btr, test_size=0.15, random_state=42)

lgbm_cls = lgb.LGBMClassifier(
    n_estimators=1000,
    learning_rate=0.05,
    num_leaves=31,          # key LightGBM param: max leaves per tree (not depth!)
    max_depth=-1,           # -1 = no limit (leaves control complexity)
    subsample=0.8,
    colsample_bytree=0.8,
    reg_alpha=0.1,
    reg_lambda=1.0,
    random_state=42,
    n_jobs=-1,
    verbose=-1              # suppress training output
)

lgbm_cls.fit(
    X_btr3, y_btr3,
    eval_set=[(X_bval3, y_bval3)],
    callbacks=[lgb.early_stopping(50, verbose=False), lgb.log_evaluation(0)]
)

proba_lgb = lgbm_cls.predict_proba(X_bte)[:, 1]
print(f"LightGBM — Best iteration: {lgbm_cls.best_iteration_}")
print(f"LightGBM — Test AUC:       {roc_auc_score(y_bte, proba_lgb):.4f}")

# Compare all four algorithms
print("\n--- Final comparison on breast cancer ---")
from sklearn.ensemble import GradientBoostingClassifier, AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
results = []
for name, model in [
    ('AdaBoost',   AdaBoostClassifier(n_estimators=200, random_state=42)),
    ('sklearn GBM', GradientBoostingClassifier(n_estimators=200, learning_rate=0.1, max_depth=3, random_state=42)),
    ('XGBoost',    xgb_cls),
    ('LightGBM',   lgbm_cls),
]:
    if name in ('XGBoost', 'LightGBM'):
        auc = roc_auc_score(y_bte, model.predict_proba(X_bte)[:, 1])
    else:
        model.fit(X_btr, y_btr)
        auc = roc_auc_score(y_bte, model.predict_proba(X_bte)[:, 1])
    print(f"  {name:14s}: AUC = {auc:.4f}")

7 The Learning Rate — Shrinkage

The learning rate (also called shrinkage) is the single most important boosting hyperparameter. It scales each tree's contribution to the ensemble:

F_t(x) = F_{t-1}(x) + η · h_t(x)

A small η forces the model to take many small steps, which is slower but generalises better. A large η takes bigger steps and converges faster but is more prone to overfitting.

In [6]:

import xgboost as xgb
from sklearn.model_selection import cross_val_score
import numpy as np

print("Learning rate vs n_estimators tradeoff (XGBoost on breast cancer):")
print(f"{'lr':>8}  {'n_est':>8}  {'CV-AUC':>10}  {'Training time'}")
print("-" * 55)

import time
configs = [
    (0.3, 50),
    (0.1, 200),
    (0.05, 400),
    (0.01, 2000),  # slow — skip if time is critical
]

for lr, n_est in configs[:3]:  # first 3 for demo
    t0 = time.time()
    m = xgb.XGBClassifier(n_estimators=n_est, learning_rate=lr, max_depth=3,
                           subsample=0.8, n_jobs=-1, random_state=42)
    auc = cross_val_score(m, X_bc, y_bc, cv=5, scoring='roc_auc').mean()
    elapsed = time.time() - t0
    print(f"{lr:>8.3f}  {n_est:>8}  {auc:>10.4f}  {elapsed:>10.2f}s")

# lr=0.300  n_est=  50: CV-AUC = 0.9821  0.45s
# lr=0.100  n_est= 200: CV-AUC = 0.9858  1.23s
# lr=0.050  n_est= 400: CV-AUC = 0.9867  2.41s
# Rule of thumb: smaller lr → better generalization, but more trees needed

8 Early Stopping

Early stopping monitors a validation metric after each tree and stops training when it hasn't improved for early_stopping_rounds consecutive rounds. This prevents overfitting and eliminates the need to grid-search n_estimators.

In [7]:

import xgboost as xgb
import lightgbm as lgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
import numpy as np

# XGBoost early stopping
xgb_es = xgb.XGBClassifier(
    n_estimators=2000,           # deliberately large
    learning_rate=0.05,
    max_depth=4,
    subsample=0.8,
    colsample_bytree=0.8,
    early_stopping_rounds=50,    # stop after 50 no-improvement rounds
    eval_metric='auc',
    n_jobs=-1, random_state=42
)
xgb_es.fit(
    X_btr2, y_btr2,
    eval_set=[(X_bval, y_bval)],
    verbose=False
)
print(f"XGBoost early stopping:")
print(f"  Stopped at round:    {xgb_es.best_iteration}")
print(f"  Best val AUC:        {xgb_es.best_score:.4f}")
print(f"  Test AUC:            {roc_auc_score(y_bte, xgb_es.predict_proba(X_bte)[:,1]):.4f}")

# LightGBM early stopping
lgbm_es = lgb.LGBMClassifier(
    n_estimators=2000,
    learning_rate=0.05,
    num_leaves=31,
    n_jobs=-1, random_state=42, verbose=-1
)
lgbm_es.fit(
    X_btr2, y_btr2,
    eval_set=[(X_bval, y_bval)],
    callbacks=[lgb.early_stopping(50, verbose=False), lgb.log_evaluation(0)]
)
print(f"\nLightGBM early stopping:")
print(f"  Best iteration:      {lgbm_es.best_iteration_}")
print(f"  Test AUC:            {roc_auc_score(y_bte, lgbm_es.predict_proba(X_bte)[:,1]):.4f}")

🌍

Real-World Spotlight: Credit Risk Modeling

Predicting loan default (probability of default, PD) is a core function of retail banking. Regulators require explainability; the business requires accuracy. Gradient boosting models dominate this space because they handle mixed data types, missing values, and nonlinear interactions — all common in credit data — without extensive preprocessing.

In [8]:

import numpy as np
import pandas as pd
import xgboost as xgb
import lightgbm as lgb
from sklearn.ensemble import GradientBoostingClassifier, AdaBoostClassifier
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
from sklearn.metrics import roc_auc_score, classification_report
from sklearn.preprocessing import StandardScaler

np.random.seed(42)
n = 20000

# Synthetic credit dataset
df = pd.DataFrame({
    'credit_utilization':   np.clip(np.random.beta(2, 5, n), 0, 1),
    'payment_history':      np.random.randint(0, 25, n),  # months on-time
    'debt_to_income':       np.random.exponential(0.35, n).clip(0, 2),
    'num_credit_lines':     np.random.poisson(3, n),
    'credit_age_months':    np.random.randint(6, 240, n),
    'num_hard_inquiries':   np.random.poisson(1.5, n).astype(int),
    'employment_years':     np.random.exponential(5, n).clip(0, 30),
    'loan_amount':          np.random.lognormal(9.5, 0.7, n),
})
# Add 15% missing values to simulate real-world data
for col in ['debt_to_income', 'employment_years']:
    mask = np.random.rand(n) < 0.15
    df.loc[mask, col] = np.nan

# Realistic default probability
default_prob = (
    0.35 * df['credit_utilization'] +
    0.25 * (1 - df['payment_history'].clip(0, 24) / 24) +
    0.20 * df['debt_to_income'].fillna(0.5) +
    0.10 * (df['num_hard_inquiries'] / 5) +
    0.10 * (1 - df['credit_age_months'].clip(6, 120) / 120)
)
df['default'] = (np.random.rand(n) < default_prob * 0.4).astype(int)
print(f"Default rate: {df['default'].mean():.1%}")

feature_cols = [c for c in df.columns if c != 'default']
X_cr = df[feature_cols].values
y_cr = df['default'].values

# XGBoost handles NaN natively — no imputation needed
Xcr_tr, Xcr_te, ycr_tr, ycr_te = train_test_split(
    X_cr, y_cr, test_size=0.2, stratify=y_cr, random_state=42)
Xcr_tr2, Xcr_val, ycr_tr2, ycr_val = train_test_split(
    Xcr_tr, ycr_tr, test_size=0.15, random_state=42)

xgb_credit = xgb.XGBClassifier(
    n_estimators=2000,
    learning_rate=0.05,
    max_depth=5,
    subsample=0.8,
    colsample_bytree=0.8,
    scale_pos_weight=(y_cr == 0).sum() / (y_cr == 1).sum(),  # handle imbalance
    reg_alpha=0.1,
    reg_lambda=1.0,
    early_stopping_rounds=50,
    eval_metric='auc',
    n_jobs=-1, random_state=42
)
xgb_credit.fit(
    Xcr_tr2, ycr_tr2,
    eval_set=[(Xcr_val, ycr_val)],
    verbose=False
)

proba_credit = xgb_credit.predict_proba(Xcr_te)[:, 1]
print(f"\nXGBoost Credit Model:")
print(f"  Best iteration: {xgb_credit.best_iteration}")
print(f"  Test AUC:       {roc_auc_score(ycr_te, proba_credit):.4f}")

# Feature importances
imp_dict = dict(zip(feature_cols, xgb_credit.feature_importances_))
print("\nFeature importances:")
for feat, imp in sorted(imp_dict.items(), key=lambda x: -x[1]):
    bar = '█' * int(imp * 100)
    print(f"  {feat:25s}: {imp:.4f} {bar}")

# Compare algorithms
print("\n--- Algorithm Comparison (5-fold CV AUC) ---")
for name, model in [
    ('AdaBoost',    AdaBoostClassifier(n_estimators=200, random_state=42)),
    ('sklearn GBM', GradientBoostingClassifier(n_estimators=200, learning_rate=0.1, max_depth=3, random_state=42)),
]:
    from sklearn.impute import SimpleImputer
    from sklearn.pipeline import Pipeline
    pipe = Pipeline([('imp', SimpleImputer(strategy='median')), ('model', model)])
    auc = cross_val_score(pipe, X_cr, y_cr, cv=5, scoring='roc_auc').mean()
    print(f"  {name:14s}: CV-AUC = {auc:.4f}")

XGBoost with early stopping achieved the best AUC, and its native missing value handling meant no imputation was needed for the 15% missing values in debt-to-income and employment years. The top predictors aligned with credit domain knowledge: credit utilization and payment history dominate. In production, models like this are monitored monthly for data drift in these top features — if credit_utilization distribution shifts (e.g., during a recession), the model performance can degrade silently.

✍️ Practice Exercises

Train AdaBoostClassifier with n_estimators=200 on the Titanic dataset. Use staged_predict_proba to plot AUC vs number of estimators. At what point does it plateau?
Compare GradientBoostingClassifier(learning_rate=0.1, n_estimators=200) vs (learning_rate=0.01, n_estimators=2000) on the Breast Cancer dataset. Which achieves better test AUC?
Train an XGBoost model on the California Housing dataset for regression (XGBRegressor). Use early stopping with a validation set. What is the best RMSE?
Try LightGBM on any large dataset (>50k rows). Compare training time vs XGBoost — how much faster is LightGBM in your environment?

📚 Primary Source for This Lesson

XGBoost Documentation: Introduction to Boosted Trees
The official XGBoost tutorial is unusually clear and mathematical — it explains exactly how XGBoost differs from standard gradient boosting. Also recommended: Chen & Guestrin (2016) "XGBoost: A Scalable Tree Boosting System" (KDD 2016) — the original paper that introduced the algorithm. For LightGBM: Ke et al. (2017) "LightGBM: A Highly Efficient Gradient Boosting Decision Tree."

💬 Getting worse results with XGBoost than a simple Random Forest? Confused about the learning rate vs n_estimators tradeoff? Paste your code and results — your tutor will help you tune the right hyperparameters.