Random Forests

A random forest fixes the decision tree's one serious flaw - high variance, a single tree that memorizes its bootstrap of noise - by averaging many of them. Each tree is trained on a random sample of the rows and sees a random subset of the features at each split; the forest's prediction is the average (regression) or majority vote (classification) across the trees. The individual trees are weak and uncorrelated-ish, and the average of many weak, diverse models is a strong, stable one. Random forests are the default "just works" model for tabular data for a reason.

How It Works

• Row sampling (bagging): each tree trains on a bootstrap sample - rows drawn with replacement, so ~63% of the data per tree. The remaining ~37% ("out-of-bag") is an implicit validation set for each tree.
• Feature randomness: at every split, only a random subset of features is considered (sqrt of the count for classification, one third for regression - the defaults). This decorrelates the trees: if every tree could pick the same strongest feature at every split, they would all look alike and the average would not help.
• Averaging: the forest's prediction is the aggregate, which cancels individual trees' errors while keeping their shared signal.

Fitting and Inspecting

import pandas as pd
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = make_classification(n_samples=2000, n_features=15, n_informative=6,
                           random_state=5)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

forest = RandomForestClassifier(n_estimators=300, random_state=0).fit(Xtr, ytr)
print("train acc:", round(forest.score(Xtr, ytr), 3))
print("test  acc:", round(accuracy_score(yte, forest.predict(Xte)), 3))
print("oob-style: fit with oob_score=True for a built-in validation number")

imp = pd.Series(forest.feature_importances_,
                index=[f"f{i}" for i in range(15)]).sort_values(ascending=False)
print(imp.round(3).head(6))

Two inspection tools matter. feature_importances_ averages the impurity reduction each feature contributes across all trees - a more stable ranking than a single tree's, but still biased toward high-cardinality features, so treat it as a screening signal, not a verdict. Fitting with oob_score=True reports the out-of-bag accuracy - each row is scored only by the trees that did not see it, a free internal validation that tracks the true test performance closely.

The Hyperparameters

The honest summary of forest tuning: the defaults are already strong, and the biggest wins come from data quality (features, cleaning) rather than from the knobs. A typical sweep is n_estimators {200, 400}, max_features {0.3, 0.5, 1.0}, max_depth {None, 10, 20} - small enough to run in minutes with cross-validation.

When Forests Win (and Lose)

• Win: medium-to-large tabular datasets, mixed feature types, moderate signal with noise, and a need for a robust baseline without heavy tuning. They are the model to beat before anything fancier.
• Lose to gradient boosting (next-but-one chapter) on many tabular problems - boosting usually edges forests by a point or two of accuracy.
• Lose on interpretability to a single pruned tree - 300 trees are an ensemble, and their explanation is the importance ranking, not a rule set.
• Lose on very small data where a well-pruned single tree or a linear model generalizes better.

Summary

Run RandomForestClassifier (or Regression) with 200-500 trees and n_jobs=-1 as the strong tabular baseline: bagging plus per-split feature randomness turns many unstable trees into one stable model. Check oob_score during development, read feature_importances_ as a screening signal, and tune max_features/max_depth lightly - the forest's power is in the averaging, not in the configuration.