Ensemble Learning

Ensemble Learning combines several models (learners) to produce predictions that are more accurate and robust than a single model.

Why Ensembles?

• Reduce variance (bagging)
• Reduce bias (boosting)
• Improve model stability
• Often wins ML competitions

Bagging (Bootstrap Aggregating)

Bagging reduces variance by training models in parallel on different subsets of the data.

Random Forest as Bagging

from sklearn.ensemble import RandomForestClassifier, BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score

iris = load_iris()
X, y = iris.data, iris.target

# Single Decision Tree
tree = DecisionTreeClassifier(random_state=42)
scores_tree = cross_val_score(tree, X, y, cv=5)

# Bagging
bagging = BaggingClassifier(
    estimator=DecisionTreeClassifier(),
    n_estimators=100,
    random_state=42
)
scores_bag = cross_val_score(bagging, X, y, cv=5)

# Random Forest
rf = RandomForestClassifier(n_estimators=100, random_state=42)
scores_rf = cross_val_score(rf, X, y, cv=5)

print(f"Decision Tree: {scores_tree.mean():.4f}")
print(f"Bagging:       {scores_bag.mean():.4f}")
print(f"Random Forest: {scores_rf.mean():.4f}")

Boosting

Boosting trains models sequentially, where each subsequent model corrects the errors of the previous one.

AdaBoost

from sklearn.ensemble import AdaBoostClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score

iris = load_iris()
X, y = iris.data, iris.target

adaboost = AdaBoostClassifier(
    n_estimators=100,
    learning_rate=0.1,
    random_state=42
)

scores = cross_val_score(adaboost, X, y, cv=5)
print(f"AdaBoost: {scores.mean():.4f} (+/- {scores.std():.4f})")

Gradient Boosting

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score

iris = load_iris()
X, y = iris.data, iris.target

gb = GradientBoostingClassifier(
    n_estimators=100,
    learning_rate=0.1,
    max_depth=3,
    random_state=42
)

scores = cross_val_score(gb, X, y, cv=5)
print(f"Gradient Boosting: {scores.mean():.4f}")

Stacking

Stacking uses a meta-learner to combine the predictions of various base models.

from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score

iris = load_iris()
X, y = iris.data, iris.target

estimators = [
    ('rf', RandomForestClassifier(n_estimators=100, random_state=42)),
    ('svm', SVC(probability=True)),
    ('lr', LogisticRegression(max_iter=200))
]

stacking = StackingClassifier(
    estimators=estimators,
    final_estimator=LogisticRegression()
)

scores = cross_val_score(stacking, X, y, cv=5)
print(f"Stacking: {scores.mean():.4f}")

Voting Classifier

from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score

iris = load_iris()
X, y = iris.data, iris.target

voting = VotingClassifier(
    estimators=[
        ('lr', LogisticRegression(max_iter=200)),
        ('rf', RandomForestClassifier(n_estimators=100)),
        ('svm', SVC(probability=True))
    ],
    voting='soft'
)

scores = cross_val_score(voting, X, y, cv=5)
print(f"Voting: {scores.mean():.4f}")

Method Comparison

When to Use Which?

• Random Forest: A good baseline, when interpretation matters
• Gradient Boosting: When high accuracy is needed
• XGBoost: Large datasets, ML competitions
• Stacking: When maximum performance is required
• Voting: A quick improvement from several models

Next chapter: Gradient Boosting, a deep dive into one of the most powerful boosting methods.

Next: Gradient Boosting →