Skillversitas

Case Study: Customer Churn

Putting the full workflow together on a churn problem: a telecom-style dataset where the target is whether a customer cancels in the next month. The steps are the tutorial in one place - load, inspect, encode, scale, split, train, tune, evaluate, set a cost-aware threshold, and report - and they transfer to any tabular binary problem.

The Data

import numpy as np

rng = np.random.default_rng(123)
n = 5000
tenure_months = rng.integers(1, 60, n)
monthly_charges = rng.normal(65, 15, n).clip(30, 120)
support_tickets = rng.poisson(1.2, n)
contract = (rng.random(n) < 0.4).astype(int)   # 1 = term contract

# churn risk: short tenure, many tickets, high charges, no term contract
z = (-0.06 * tenure_months - 1.5 * support_tickets
     + 0.03 * (monthly_charges - 65) - 0.8 * contract)
p = 1 / (1 + np.exp(-z))
y = (rng.random(n) < p).astype(int)

X = np.c_[tenure_months, monthly_charges, support_tickets, contract]
print("churn rate:", round(y.mean(), 3))   # ~28% - mildly imbalanced

Train and Evaluate

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report, roc_auc_score

Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

pipe = Pipeline([("scaler", StandardScaler()),
                 ("model", LogisticRegression(max_iter=3000))])

cv = cross_val_score(pipe, Xtr, ytr, cv=StratifiedKFold(5), scoring="roc_auc")
print("CV AUC:", round(cv.mean(), 4), "+/-", round(cv.std(), 4))

pipe.fit(Xtr, ytr)
proba = pipe.predict_proba(Xte)[:, 1]
print("test AUC:", round(roc_auc_score(yte, proba), 4))
print(classification_report(yte, (proba >= 0.5).astype(int)))

The Cost-Aware Threshold

A retained customer saved by a win-back offer costs ~$40; a churned customer is worth ~$600 in lost margin. The offer is worth sending when p(churn) > 40 / (40 + 600) ≈ 0.06 - far below 0.5:

save_cost, lost_value = 40.0, 600.0
cutoff = save_cost / (save_cost + lost_value)

offer = (proba >= cutoff).astype(int)
tp = int(((offer == 1) & (yte == 1)).sum())
fp = int(((offer == 1) & (yte == 0)).sum())
print(f"offer to {offer.sum()} customers; true churners reached: {tp}; "
      f"offers wasted: {fp}")
print(f"expected value: {tp * (lost_value - save_cost) - fp * save_cost:+.0f}")

This is the whole business case in four lines: the probability model, a threshold from the economics, and an expected-value computation the offer program can be judged by. No threshold at 0.5, no accuracy number.

The Report a Stakeholder Sees

  • Performance: CV AUC with spread, test AUC, per-class precision/recall at the operating threshold.
  • Drivers: odds ratios with confidence intervals (support tickets, tenure, contract) - what the model says churn is driven by.
  • Decision: offer list size, expected saved margin, sensitivity to the cost ratio.
  • Monitoring plan: track probability-bin calibration and AUC on monthly cohorts; drift in the bins is the early warning.

Summary

The churn workflow - encode, scale in a pipeline, CV for an honest AUC, a threshold from the cost ratio, and an expected-value report - is the template for almost every production scoring model. The logistic regression is the simplest component in it; the discipline around it is the product.