Evaluating the Model
A trained model is a hypothesis; evaluation is the experiment that tests it. The discipline is simple: hold out data the model never saw, measure several metrics on that data, and only then decide whether the model is good enough. This chapter sets up the measurement loop - and warns against the most common failure, which is evaluating on data the model memorized.
The Hold-Out Set
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
X, y = make_classification(n_samples=2000, n_features=10, n_informative=5,
random_state=3)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
scaler = StandardScaler().fit(Xtr)
model = LogisticRegression(max_iter=2000).fit(scaler.transform(Xtr), ytr)
Xte_s = scaler.transform(Xte)
print("train acc:", model.score(scaler.transform(Xtr), ytr))
print("test acc:", model.score(Xte_s, yte))If train and test accuracy are close, the model generalizes. If train is much higher, it is overfitting. If both are low, the signal is weak or the model is too simple. The test set answers "how well will it do on new data?" - and it should be touched as few times as possible, because repeated peeking quietly turns it into a training set.
The Metric Menu
| Metric | Answer | Use when |
|---|---|---|
| Accuracy | What fraction is right overall | Classes balanced, errors roughly equal-cost |
| Confusion matrix | How each error type is distributed | Always - it is the raw material for the rest |
| Precision / Recall / F1 | Quality of positives; coverage of positives | Imbalanced classes or asymmetric error costs |
| ROC AUC | Ranking quality across all thresholds | Comparing models before threshold choice |
| Log loss / Brier score | Calibration of the probabilities | The probability itself is the product |
Measuring the Probabilities
from sklearn.metrics import log_loss, brier_score_loss
proba = model.predict_proba(Xte_s)[:, 1]
print("log loss:", round(log_loss(yte, proba), 4))
print("brier :", round(brier_score_loss(yte, proba), 4))Log loss is mean cross-entropy - the same objective the training minimizes - so a big gap between train and test log loss is the cleanest overfitting signal. Brier score is the squared error of the probabilities and is easier to explain to non-technical stakeholders ("on average we were off by this much in probability units").
Cross-Validation Preview
With small data a single 70/30 split is noisy: one unlucky split can flatter or sink the model. k-fold cross-validation rotates the hold-out - train on k-1 folds, test on the remaining one, repeat k times, and average - using every sample for both training and testing exactly once. The dedicated tuning chapter uses it to choose hyperparameters; for now the rule is: small dataset, use 5-fold CV instead of a single split.
Summary
Evaluate on unseen data, always report the confusion matrix plus at least one probability metric, and compare train vs test to detect overfitting. Treat the test set as a single final exam - peek at it through cross-validation during development, and only take the real exam once.