Train-Test Split

The train-test split is the single most important decision in model evaluation: it creates the data the model never sees during development, so that the final score measures generalization instead of memorization. Done once, early, with the right options, it makes every subsequent number trustworthy. Done late, or done repeatedly while tuning, it silently inflates the results and the inflation is nearly impossible to detect after the fact.

The Split Itself

from sklearn.model_selection import train_test_split

Xtr, Xte, ytr, yte = train_test_split(
    X, y,
    test_size=0.3,        # 70% train / 30% test
    random_state=0,       # reproducible split
    stratify=y,           # preserve the class distribution in both sets
)

print(len(Xtr), len(Xte))
print("train rate:", round(ytr.mean(), 3), "test rate:", round(yte.mean(), 3))

test_size=0.3 is the conventional default (more training data is usually better than a bigger test set, and 30% is enough for a stable score). stratify=y keeps the class proportions identical in both sets - essential for imbalanced problems, where a random split might leave the test set with a lucky or unlucky number of positives. random_state makes the split reproducible across runs and machines.

Once, and Early

The workflow rule: create the split before any modeling, keep the test set in a variable (or a separate file) that only the final evaluation touches, and let the entire development loop - feature work, model choice, tuning - run on the train set alone, with cross-validation inside it playing the validation role.

# the loop works only on the train set
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression

cv = cross_val_score(LogisticRegression(max_iter=2000), Xtr, ytr, cv=5)
print("CV score (development):", round(cv.mean(), 4))

# the test set appears exactly once, at the end
model = LogisticRegression(max_iter=2000).fit(Xtr, ytr)
print("test score (final):", model.score(Xte, yte))

If you check the test score, change something, and check it again, the test set is no longer measuring unseen behavior - it is measuring your search process. That is the difference between a measurement and an experiment, and it is why the split is a gate, not a convenience.

Time-Series Splits Are Different

Random splitting is wrong for data with time order: a model could learn from "future" rows and be tested on the "past." The split must respect chronology - train on the earlier period, test on the later one:

df = df.sort_values("date")
cutoff = df["date"].quantile(0.7)

train = df[df["date"] <= cutoff]
test = df[df["date"] > cutoff]     # the model predicts the future, as it will in production

For time series, "cross-validation" also changes to a rolling or expanding window that always trains before it tests - the TimeSeriesSplit object in scikit-learn does exactly that. The principle is the same: no information from the test period may reach the training side.

How Big Should Each Set Be?

Small data (< a few thousand rows): 70/30 or an 80/20 split, plus cross-validation on the train portion.
Large data (millions of rows): a smaller test set (10%) is enough for a precise score - the training benefit of the extra rows outweighs test precision.
Very imbalanced data: stratify, and check that the test set still contains enough positives for the minority metric to be meaningful (a few hundred at minimum).

Summary

Split early with train_test_split(test_size=0.3, random_state=..., stratify=y), develop entirely on the train set using cross-validation, and evaluate on the test set exactly once at the end. For time-ordered data, split by time instead of by shuffle. The split is the foundation every later number stands on - get the gate right and the rest of the workflow is honest.