Random Forest

Random Forest is an ensemble learning method that builds many decision trees and combines their predictions to improve accuracy and reduce overfitting.

Bagging Concept

Random Forest is based on Bootstrap Aggregating (Bagging):
1. Create a bootstrap sample (sampling with replacement) from the data
2. Train a decision tree on each bootstrap sample
3. Aggregate the predictions from all trees

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42
)

model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

print(f"Accuracy: {model.score(X_test, y_test):.4f}")

Feature Randomness

Each split in Random Forest only considers a random subset of features (usually √n features for classification). This reduces the correlation between trees.

Out-of-Bag (OOB) Score

The OOB score is a performance estimate that does not require a separate validation set. Each tree is evaluated using the data that was NOT included in its bootstrap sample.

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris

iris = load_iris()
X, y = iris.data, iris.target

model = RandomForestClassifier(
    n_estimators=100,
    oob_score=True,
    random_state=42
)
model.fit(X, y)

print(f"OOB Score: {model.oob_score_:.4f}")

Feature Importance

Random Forest provides feature importance based on the average impurity decrease across each tree.

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_wine
import pandas as pd

wine = load_wine()
X, y = wine.data, wine.target

model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X, y)

importances = pd.Series(model.feature_importances_, index=wine.feature_names)
importances = importances.sort_values(ascending=False)

print("Top 5 Feature Importances:")
print(importances.head())

Comparing the Number of Trees

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score

iris = load_iris()
X, y = iris.data, iris.target

n_trees = [10, 50, 100, 200, 500]
for n in n_trees:
    model = RandomForestClassifier(n_estimators=n, random_state=42)
    scores = cross_val_score(model, X, y, cv=5)
    print(f"n={n:4d} → Accuracy: {scores.mean():.4f} (+/- {scores.std():.4f})")

Example: Digit Classification

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
X_train, X_test, y_train, y_test = train_test_split(
    digits.data, digits.target, test_size=0.2, random_state=42
)

model = RandomForestClassifier(n_estimators=200, random_state=42)
model.fit(X_train, y_train)

train_acc = model.score(X_train, y_train)
test_acc = model.score(X_test, y_test)
print(f"Train Accuracy: {train_acc:.4f}")
print(f"Test Accuracy: {test_acc:.4f}")

Pros and Cons

Random Forest is a good choice for a baseline model. Next chapter: Naive Bayes, a probabilistic algorithm based on Bayes' theorem.

Next: Naive Bayes →