Overfitting & Underfitting

Overfitting and underfitting are the two most common problems in machine learning. Understanding and addressing them is essential for building generalizable models.

Bias-Variance Tradeoff

Total Error = Bias² + Variance + Irreducible Error

Underfitting

The model is too simple to capture the patterns in the data.

Signs:

• Low training accuracy
• Low test accuracy
• Model is not complex enough

Solutions:

• Add features / feature engineering
• Reduce regularization
• Use a more complex model
• Reduce noise

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.metrics import r2_score
import numpy as np

np.random.seed(42)
X = np.sort(np.random.rand(100, 1) * 10, axis=0)
y = np.sin(X).ravel() + np.random.randn(100) * 0.3

# Underfitting: linear
model_linear = LinearRegression()
model_linear.fit(X, y)
y_pred = model_linear.predict(X)
print(f"Linear - R²: {r2_score(y, y_pred):.4f}")

# Good fit: polynomial degree 3
model_poly = make_pipeline(PolynomialFeatures(3), LinearRegression())
model_poly.fit(X, y)
y_pred = model_poly.predict(X)
print(f"Poly degree 3 - R²: {r2_score(y, y_pred):.4f}")

Overfitting

The model is too complex and memorizes the noise in the training data.

Signs:

• Very high training accuracy
• Test accuracy much lower
• Large gap between train and test performance

Solutions:

• Regularization (L1/L2)
• Dropout (neural networks)
• Early stopping
• Reduce model complexity
• Add more training data
• Cross validation

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42
)

# Overfitting: no constraints
model_overfit = DecisionTreeClassifier(random_state=42)
model_overfit.fit(X_train, y_train)
print(f"Overfitting - Train: {model_overfit.score(X_train, y_train):.4f}, "
      f"Test: {model_overfit.score(X_test, y_test):.4f}")

# Good fit: with pruning
model_good = DecisionTreeClassifier(max_depth=3, random_state=42)
model_good.fit(X_train, y_train)
print(f"Good fit - Train: {model_good.score(X_train, y_train):.4f}, "
      f"Test: {model_good.score(X_test, y_test):.4f}")

Regularization

L1 Regularization (Lasso)

Adds a penalty based on the absolute value of the coefficients. Can produce a sparse model (feature selection).

from sklearn.linear_model import Lasso
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
import numpy as np

X, y = make_regression(n_samples=100, n_features=20, n_informative=5, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

lasso = Lasso(alpha=1.0)
lasso.fit(X_train, y_train)

n_zero = np.sum(lasso.coef_ == 0)
print(f"Zeroed-out features: {n_zero}/20")
print(f"Train R²: {lasso.score(X_train, y_train):.4f}")
print(f"Test R²: {lasso.score(X_test, y_test):.4f}")

L2 Regularization (Ridge)

Adds a penalty based on the square of the coefficients.

from sklearn.linear_model import Ridge

ridge = Ridge(alpha=1.0)
ridge.fit(X_train, y_train)
print(f"Ridge Train R²: {ridge.score(X_train, y_train):.4f}")
print(f"Ridge Test R²: {ridge.score(X_test, y_test):.4f}")

Dropout (Neural Network)

from tensorflow import keras

model = keras.Sequential([
    keras.layers.Dense(128, activation='relu', input_shape=(10,)),
    keras.layers.Dropout(0.5),
    keras.layers.Dense(64, activation='relu'),
    keras.layers.Dropout(0.3),
    keras.layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

Early Stopping

from tensorflow import keras

callback = keras.callbacks.EarlyStopping(
    monitor='val_loss',
    patience=5,
    restore_best_weights=True
)

# model.fit(X_train, y_train, epochs=100, 
#           validation_split=0.2, callbacks=[callback])

Learning Curve

from sklearn.model_selection import learning_curve
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
import matplotlib.pyplot as plt
import numpy as np

iris = load_iris()
model = RandomForestClassifier(n_estimators=100, random_state=42)

train_sizes, train_scores, test_scores = learning_curve(
    model, iris.data, iris.target, cv=5,
    train_sizes=np.linspace(0.1, 1.0, 10)
)

plt.figure(figsize=(8, 5))
plt.plot(train_sizes, train_scores.mean(axis=1), label='Training')
plt.plot(train_sizes, test_scores.mean(axis=1), label='Validation')
plt.xlabel('Training Size')
plt.ylabel('Score')
plt.title('Learning Curve')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

Summary

Next chapter: Ensemble Learning, combining many models for better performance.

Next: Ensemble Learning →