Feature Scaling and Preprocessing

Logistic regression is one of the models where feature scaling is not optional decoration - it changes the answer. Two reasons: the regularization penalty measures weights in whatever units the features happen to use, and gradient descent converges slowly on loss surfaces with stretched axes. A feature measured in microseconds and one measured in hours can differ by a factor of a million, and both problems follow from that.

The Two Problems

• The penalty problem: L2 penalizes w^2. If feature A is 1000x larger in scale, its weight will be ~1000x smaller to produce the same score - and the penalty then crushes it 1000^2 = a million times harder than feature B's weight. Without scaling, the regularization budget is spent on units, not on signal.
• The optimization problem: the loss becomes an elongated valley. A learning rate safe on the wide axis overshoots on the narrow one, and convergence crawls - or fails within max_iter.

StandardScaler

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)
X = np.c_[rng.normal(50_000, 15_000, 500),    # income: tens of thousands
          rng.normal(2, 1, 500)]              # score: around 2
y = (X[:, 0] / 10_000 + X[:, 1] > 1).astype(int)

Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

# without scaling: the two weights are incomparable in magnitude
raw = LogisticRegression(C=1.0, max_iter=5000).fit(Xtr, ytr)
print("unscaled coefs:", np.round(raw.coef_[0], 6))

# with scaling: the penalty now sees comparable features
scaler = StandardScaler().fit(Xtr)
scaled = LogisticRegression(C=1.0, max_iter=5000).fit(scaler.transform(Xtr), ytr)
print("scaled   coefs:", np.round(scaled.coef_[0], 3))

StandardScaler subtracts the training mean and divides by the training standard deviation, so every feature has mean 0 and variance 1. It is the default choice for logistic regression.

Other Scalers

Tree-based models do not need scaling (splits are invariant to monotone transforms), which is a common source of "why does my pipeline scale?" - for logistic regression, it is because the penalty and the optimizer both care about units.

Fit on Train Only

The scaler's mean and standard deviation are fitted on the training set and then applied - unchanged - to validation and test data. Fitting on the test set leaks its statistics into training: a small, consistent overestimate of performance that shows up as train/test gaps closing for the wrong reason. In a pipeline (next categories cover the tool) the scaler's fit happens on the train split automatically.

Other Preprocessing in the Same Step

• Encode categoricals (one-hot or target encoding) before fitting - logistic regression needs numbers.
• Decide how missing values are handled (impute, or a missingness indicator) - the scaler cannot see NaNs.
• Check for duplicate or constant features; they add nothing and can distort the penalty.

Summary

Standardize features before logistic regression: it makes the regularization penalty fair across features and keeps gradient descent on a round loss surface. Fit the scaler on train only, use RobustScaler when outliers dominate, and treat scaling as part of the model - in production it ships inside a pipeline, never as a forgotten notebook step.