Skillversitas

Handling Imbalanced Datasets

Most real classification problems are skewed: 5% of transactions are fraud, 2% of patients relapse, 30% of customers churn. Imbalance breaks the naive pipeline in two ways - accuracy becomes a meaningless metric (predicting the majority class always scores 95%), and the optimizer, minimizing average loss, can afford to ignore the minority class entirely. The fixes range from a single parameter to resampling the data, and each has costs.

The Accuracy Trap

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(2)
X = rng.normal(size=(2000, 8))
y = (rng.random(2000) < 0.05).astype(int)   # 5% positives
X[:100] += 1.2   # the positives have a weak signal

Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

model = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
print("accuracy:", model.score(Xte, yte))   # ~0.95 - and useless
print(classification_report(yte, model.predict(Xte)))

Read the report's rows, not the accuracy line: the positive class's recall is what tells you how many real cases you missed, and its precision tells you how much of your alert volume is genuine.

Reweighting the Loss

The cheapest fix is to make errors on the minority class cost more during training. scikit-learn's class_weight="balanced" multiplies each class's loss by n_samples / (n_classes * n_class_count) - with a 5% minority, its errors count 19x more:

model_bal = LogisticRegression(class_weight="balanced", max_iter=2000).fit(Xtr, ytr)
print(classification_report(yte, model_bal.predict(Xte)))
# positive recall usually jumps; precision usually drops
# the threshold still matters - see the cost-aware cutoff

Reweighting does not change the decision boundary's direction, only where it sits relative to the class masses - it is a soft version of moving the threshold, applied at training time. You can also pass explicit weights, class_weight={0: 1, 1: 10}, to encode the business cost ratio directly.

Resampling the Data

  • Undersample the majority (drop random majority rows to match the minority): faster training, same boundary math as reweighting when done correctly. Risk: throwing away information; always undersample on the training fold only.
  • Oversample the minority (duplicate rows): equivalent to reweighting for linear models, but for nonlinear models it can overfit the duplicated points.
  • Synthetic - SMOTE: interpolate new minority points between existing neighbors. It changes the data geometry, helps some tree/NN models, and can create implausible combinations; validate with CV that it actually helps before adopting it.

The Discipline

  • Stratify every split so each fold keeps the real 5/95 ratio - otherwise one fold can be lucky or unlucky by chance.
  • Resample inside each CV fold, never on the whole dataset before splitting (leakage).
  • Report per-class metrics and a precision-recall view; ROC-AUC alone can look fine while precision is terrible.
  • Set the threshold from the cost ratio on the validation set, after - not before - reweighting.

Summary

Start with class_weight="balanced" (or explicit business-cost weights) and stratified splits; add SMOTE only if cross-validation proves it helps; and evaluate with per-class precision/recall, never accuracy. On rare-event problems the minority class's recall and precision are the entire report card.