Skillversitas

Decision Trees

A decision tree learns a sequence of if-then questions about the features, splitting the data at each node until the remaining points are (approximately) one class. The result is a model you can literally read: "if balance > 500 and age < 30 then high-risk, else low-risk." That readability, plus zero need for scaling and native handling of mixed feature types, makes the tree the most interpretable non-linear model in the standard toolkit - and its tendency to overfit is what random forests exist to fix.

How It Splits

At each node the tree tries every feature and every threshold, and keeps the split that most reduces impurity. The two impurity measures are Gini (the default) and entropy - both quantify "how mixed are the classes here," and both push the tree toward pure leaves. The stopping rules are the hyperparameters: max_depth (how many questions deep), min_samples_leaf (the minimum points per leaf), and max_features (considered at each split - in trees, usually all of them).

Fitting and Inspecting

import numpy as np
from sklearn.datasets import make_classification
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = make_classification(n_samples=1000, n_features=8, n_informative=4,
                           random_state=3)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

tree = DecisionTreeClassifier(max_depth=4, random_state=0).fit(Xtr, ytr)
print("train acc:", round(tree.score(Xtr, ytr), 3))
print("test  acc:", round(accuracy_score(yte, tree.predict(Xte)), 3))

print(export_text(tree, feature_names=[f"f{i}" for i in range(8)]))
print("importances:", np.round(tree.feature_importances_, 3))

Three things to read from the output. The train/test gap: an unpruned tree scores ~1.0 on train and well below that on test - the overfitting signature. The exported text: the actual decision rules, the model's explanation. The feature importances: how much each feature reduced impurity across the tree - a first-pass variable selection, with the caveat that correlated features split the credit.

The Overfitting Problem and Its Fixes

Left alone, a tree grows until every leaf is pure - it memorizes the training set. The controls, in order of use:

  • max_depth - the coarsest and most effective; depth 3-6 for most tabular problems.
  • min_samples_leaf - leaves must contain at least this many points; a strong regularizer that produces stumpy, readable trees.
  • ccp_alpha - cost-complexity pruning: grow a big tree, then prune subtrees whose removal improves the regularized score.
  • Cross-validation over the controls - the depth/leaf combination that maximizes CV score is the one to ship.

When Trees Are the Right Tool

  • The audience needs to see the rules - trees are the model you can put in a slide deck and defend.
  • Features are mixed (numeric plus categorical) - no encoding or scaling required.
  • Interactions and non-monotone effects exist - trees find them without engineering.
  • A single tree is too unstable - step up to random forests (next chapter), which keep the interpretability and kill the variance.

Where trees struggle: smooth relationships with noise (they fragment into steps), very high dimensions (each split sees one feature), and extrapolation beyond the training range (a tree's prediction is flat outside its observed range - it cannot trend).

Summary

Fit DecisionTreeClassifier with an explicit max_depth (and min_samples_leaf), read the rules with export_text, and check feature importances as a first-pass signal ranking. Expect the classic overfit signature - perfect train score, weaker test score - and control it with the pruning parameters chosen by cross-validation. A pruned tree is the most readable non-linear model you will deploy; an unpruned one is a memorization machine with a chart attached.