Bagging, Boosting and Stacking

Ensembles combine multiple models because uncorrelated errors average out. The three standard strategies differ in one axis - how the base models are trained: in parallel on resampled data (bagging), in sequence on the mistakes of their predecessors (boosting), or in parallel plus a final meta-model that learns the blend (stacking). Random forests, already covered, are the bagging case; this chapter covers boosting and stacking, where the accuracy gains on tabular data usually come from.

Boosting: Learning from Mistakes

Gradient boosting trains weak learners - usually shallow trees - sequentially: each new tree fits the residuals of the current ensemble, so the sum of all trees approaches the target from different directions. The learning_rate shrinks each tree's contribution, letting you take many small, well-regularized steps instead of few large ones.

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score

gb = GradientBoostingClassifier(
    n_estimators=400,
    learning_rate=0.05,
    max_depth=3,
    subsample=0.8,
    random_state=0,
)
scores = cross_val_score(gb, Xtr, ytr, cv=5, scoring="roc_auc")
print("CV AUC:", round(scores.mean(), 4), "+/-", round(scores.std(), 4))

The tuning logic is the mirror image of bagging: many small steps beat few big ones. A working recipe is learning_rate between 0.01 and 0.1, n_estimators as high as the compute budget allows (stop when CV plateaus), max_depth between 2 and 5, and subsample below 1.0 for noise robustness. External libraries - XGBoost, LightGBM, CatBoost - implement the same idea faster with more controls, and they drop into scikit-learn's pipeline and search APIs unchanged.

Stacking: Blending Different Models

Stacking trains several diverse base models, then trains a meta-learner on their out-of-fold predictions - so the blender learns which model to trust on which kind of row:

from sklearn.ensemble import StackingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.metrics import roc_auc_score

stack = StackingClassifier(
    estimators=[
        ("lr", LogisticRegression(max_iter=2000)),
        ("rf", RandomForestClassifier(n_estimators=200, random_state=0)),
        ("svm", SVC(probability=True, random_state=0)),
    ],
    final_estimator=LogisticRegression(),
    cv=5,
)
stack.fit(Xtr, ytr)
proba = stack.predict_proba(Xte)[:, 1]
print("test AUC:", round(roc_auc_score(yte, proba), 4))

The internal cv is what makes stacking honest: the base-model predictions used to train the meta-learner are generated on held-out folds, not on the rows those base models were fitted on - otherwise the meta-learner just memorizes confident training outputs. Stacking wins when the base models make different kinds of mistakes; it costs extra training time and adds a layer that is hard to explain.

Choosing Among Them

A practical order: tune a random forest, then a gradient booster on the same pipeline; if the booster wins by more than its own noise, ship it. Stacking is the escalation step for when several diverse models each capture something the others miss.

Summary

Bagging averages diverse models trained in parallel; boosting fits each new weak learner to the ensemble's remaining errors; stacking adds a meta-learner over out-of-fold predictions. For tabular accuracy, gradient boosting is the default strong model - small learning rate, many shallow trees, subsampling for noise - with XGBoost or LightGBM as the better-engineered versions of the same idea. Ship the simplest model that beats the rest by more than its own noise, and let the CV spread, not the single best cell, decide when to stop.