Skillversitas

Model Evaluation and Cross-Validation

Evaluation is the step that separates a result from a claim: it measures how well the model generalizes, not how well it memorized. The tools are the metric suite (accuracy and its imbalanced-data corrections for classification; R2 and error metrics for regression) plus cross-validation, which turns a single lucky-or-unlucky split into a stable estimate by rotating the hold-out across the whole dataset. This chapter is the measurement toolkit the tuning chapter builds on.

Classification Metrics

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import (accuracy_score, precision_score, recall_score,
                             f1_score, roc_auc_score, confusion_matrix)

X, y = make_classification(n_samples=1500, weights=[0.9, 0.1],
                           n_informative=5, random_state=2)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
model = LogisticRegression(max_iter=2000, class_weight="balanced").fit(Xtr, ytr)
proba = model.predict_proba(Xte)[:, 1]
pred = model.predict(Xte)

print("accuracy:", round(accuracy_score(yte, pred), 3))
print("precision:", round(precision_score(yte, pred), 3))
print("recall   :", round(recall_score(yte, pred), 3))
print("f1       :", round(f1_score(yte, pred), 3))
print("ROC AUC  :", round(roc_auc_score(yte, proba), 3))
print(confusion_matrix(yte, pred))

The confusion matrix is the raw material; every ratio is a view of it. On imbalanced data, accuracy lies (predict the majority and score 90%), so the report is precision/recall/F1 for the minority class plus ROC AUC as the threshold-free ranking score. The choice among them is the business decision: precision when false alarms are costly, recall when misses are costly, AUC when you want the model's ranking quality independent of any threshold.

Regression Metrics

from sklearn.metrics import r2_score, mean_absolute_error, mean_squared_error

pred = model_reg.predict(Xte)
print("R2:", round(r2_score(yte, pred), 4))
print("MAE:", round(mean_absolute_error(yte, pred), 4))
print("RMSE:", round(mean_squared_error(yte, pred, squared=False), 4))

R2 is the variance-explained fraction (1.0 perfect, 0.0 the mean is as good as the model, negative means worse than the mean). MAE is the average error in target units - the metric to quote to a stakeholder ("we are off by about X on average"). RMSE squares the errors first, so it penalizes the big misses harder - the right metric when a single large error is more damaging than several small ones. Report all three: they answer different questions about the same prediction.

Cross-Validation

from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(5, shuffle=True, random_state=0)
scores = cross_val_score(LogisticRegression(max_iter=2000), Xtr, ytr,
                         cv=cv, scoring="roc_auc")
print("CV AUC:", round(scores.mean(), 4), "+/-", round(scores.std(), 4))

k-fold cross-validation splits the data into k parts, trains on k-1 and validates on the remaining one, k times, and averages - so every row serves as validation exactly once. StratifiedKFold keeps the class ratio in every fold (essential for imbalanced data); plain KFold is for regression; TimeSeriesSplit respects chronological order. The mean is the estimate; the standard deviation is the stability signal - a large spread means the score depends on which rows landed where, and the model or the data is noisier than the point estimate suggests.

The Evaluation Discipline

  • CV during development, held-out test at the end. Cross-validation answers "how good is this configuration?"; the final test set answers "how good is this model?" - one number, produced once.
  • The metric must match the decision. If the business cost is asymmetric, the evaluation metric is asymmetric too (F-beta with beta matching the cost ratio, or a direct cost computation).
  • Compare against the trivial baseline. "Always predict the majority class" or "always predict the mean" is the floor; a model that cannot beat it is not a model, it is noise.
  • Report the spread, not just the mean. A CV score without its standard deviation is an unqualified claim.

Summary

Measure with the full suite - confusion matrix plus precision/recall/F1 and AUC for classification, R2/MAE/RMSE for regression - and estimate stability with stratified (or time-aware) cross-validation rather than a single split. Match the metric to the business cost, beat the trivial baseline, quote the mean with its spread, and reserve one untouched test set for the final, single, definitive number. Evaluation done this way is the difference between a notebook result and a defensible claim.