Model Evaluation and Metrics

Fitting a model is easy; proving it will work is not. Evaluation answers three questions: how accurate is the model on unseen data, which errors does it make, and which settings of the model are best. Choosing honest metrics and resampling schemes prevents the most common failure in applied machine learning — a model that looks great in the notebook and fails in production.

Train, Validation, and Test Splits

Data must be partitioned before any evaluation. The test set is touched exactly once, at the very end. Stratify classification targets so minority classes keep their share, and never fit transformers or feature selectors on the full dataset first.

from sklearn.model_selection import train_test_split

# 80% fit/tune, 20% held out for final judging
X_tr, X_tmp, y_tr, y_tmp = train_test_split(X, y, test_size=0.3,
                                            stratify=y, random_state=42)
X_val, X_test, y_val, y_test = train_test_split(X_tmp, y_tmp, test_size=0.5,
                                                stratify=y_tmp, random_state=42)

Cross-Validation

A single split can be lucky or unlucky. k-fold cross-validation rotates the hold-out role over k folds and averages the scores, giving a mean and a standard deviation that reflect real variability.

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
model = RandomForestClassifier(n_estimators=300, random_state=42)
scores = cross_val_score(model, X_tr, y_tr, cv=cv, scoring="f1_macro")
print("F1: %.3f +/- %.3f" % (scores.mean(), scores.std()))

Classification Metrics

Accuracy hides imbalance: predicting "no fraud" for everyone scores 99.8% and finds zero fraud. The confusion matrix exposes the four error types, precision measures how many positive predictions were right, recall measures how many true positives were caught, and F1 balances the two. The ROC AUC measures ranking quality across all thresholds.

from sklearn.metrics import (confusion_matrix, precision_score,
                             recall_score, f1_score, roc_auc_score,
                             classification_report)

pred = model.fit(X_tr, y_tr).predict(X_val)
print(confusion_matrix(y_val, pred))
print(precision_score(y_val, pred), recall_score(y_val, pred), f1_score(y_val, pred))
proba = model.predict_proba(X_val)[:, 1]
print("ROC AUC:", roc_auc_score(y_val, proba))
print(classification_report(y_val, pred))

Regression Metrics

R² is the default headline, but MAE communicates in the target's own units ("off by 12,000 on average"), and RMSE punishes large misses more. Compare RMSE against the mean of the target so the number means something practical.

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

pred = reg.predict(X_test)
print("MAE :", mean_absolute_error(y_test, pred))
print("RMSE:", np.sqrt(mean_squared_error(y_test, pred)))
print("R2  :", r2_score(y_test, pred))

Hyperparameter Tuning

Tuning happens on the validation data, never on the test set. GridSearchCV tries every combination; RandomizedSearchCV samples the space and scales better. Score with a pipeline included so preprocessing is refit inside every fold.

from sklearn.model_selection import GridSearchCV

grid = GridSearchCV(model, {"max_depth": [6, 10, None],
                            "min_samples_leaf": [1, 3, 10]},
                    cv=cv, scoring="f1_macro", n_jobs=-1)
grid.fit(X_tr, y_tr)
print(grid.best_params_, grid.best_score_)

Summary

Sound evaluation separates train, validation, and test data, uses cross-validation to estimate stable scores, and selects metrics that match the problem — confusion matrix, precision/recall/F1, and ROC AUC for classification; MAE, RMSE, and R² for regression. Tune hyperparameters only on validation signals and read the test set once, when the final model is frozen.