Decision Tree

A Decision Tree is a supervised learning algorithm that works by splitting the data into subsets based on features. It is highly intuitive and easy to visualize.

How It Works

A Decision Tree makes decisions in the form of a tree:
- Root Node: The first feature tested
- Internal Node: The next feature to test
- Leaf Node: The resulting prediction label/class

Gini Impurity

Gini impurity measures the impurity of a node. The lower the Gini, the purer the node.

Gini = 1 - Σ(pi²)

Where pi is the proportion of class i within the node.

Information Gain & Entropy

Entropy measures uncertainty:

Entropy = -Σ(pi × log2(pi))

Information Gain = Entropy of parent - Σ(weighted) Entropy of children.

import numpy as np

def gini_impurity(labels):
    classes = np.unique(labels)
    gini = 1
    for c in classes:
        p = np.sum(labels == c) / len(labels)
        gini -= p ** 2
    return gini

# Example
node1 = np.array([0, 0, 0, 1, 1])  # low Gini
node2 = np.array([0, 1, 0, 1, 0])  # high Gini

print(f"Node 1 Gini: {gini_impurity(node1):.4f}")
print(f"Node 2 Gini: {gini_impurity(node2):.4f}")

Implementation with Scikit-Learn

from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42
)

model = DecisionTreeClassifier(
    max_depth=3,
    criterion='gini',
    random_state=42
)
model.fit(X_train, y_train)

print(f"Accuracy: {model.score(X_test, y_test):.4f}")

# Tree as text
tree_rules = export_text(model, feature_names=iris.feature_names)
print(tree_rules)

Visualizing a Decision Tree

from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.datasets import load_iris
import matplotlib.pyplot as plt

iris = load_iris()
model = DecisionTreeClassifier(max_depth=3, random_state=42)
model.fit(iris.data, iris.target)

plt.figure(figsize=(15, 8))
plot_tree(model,
          feature_names=iris.feature_names,
          class_names=list(iris.target_names),
          filled=True,
          rounded=True)
plt.title("Decision Tree Iris Dataset")
plt.tight_layout()
plt.show()

Pruning

Pruning prevents overfitting by limiting the depth of the tree.

from sklearn.tree import DecisionTreeClassifier

# Without pruning - prone to overfitting
model_full = DecisionTreeClassifier(random_state=42)
model_full.fit(X_train, y_train)
print(f"Without pruning - Train: {model_full.score(X_train, y_train):.4f}, "
      f"Test: {model_full.score(X_test, y_test):.4f}")

# With pruning
model_pruned = DecisionTreeClassifier(max_depth=3, random_state=42)
model_pruned.fit(X_train, y_train)
print(f"With pruning - Train: {model_pruned.score(X_train, y_train):.4f}, "
      f"Test: {model_pruned.score(X_test, y_test):.4f}")

Feature Importance

A Decision Tree provides information about how important each feature is:

import pandas as pd

importances = model.feature_importances_
features = iris.feature_names

df_imp = pd.DataFrame({
    'Feature': features,
    'Importance': importances
}).sort_values('Importance', ascending=False)

print(df_imp)

Pros and Cons

Decision Trees tend to overfit, which motivates the use of ensemble methods. Next chapter: Random Forest.

Next: Random Forest →