Regularization (L1 and L2)

Without a penalty, the loss keeps decreasing as the weights grow larger - pushing confident predictions further from the boundary always helps the training set. Regularization adds a cost on the weight magnitudes, trading a little training accuracy for smaller, more stable weights that generalize better and stay interpretable. The two standard penalties are L2 (shrink everything) and L1 (shrink some features to exactly zero).

L2 (Ridge-Style)

The penalty is lambda times the squared norm of the weights. Its gradient is a constant multiple of the weights themselves, so every update contains a small pull of each weight toward zero - a smooth shrinkage that keeps all features but damps the noisy ones:

loss = log_loss + lambda * ||w||^2
gradient = X^T(p - y)/n + 2 * lambda * w

Practical effects: more stable coefficients when features are correlated, better behavior with more features than samples, and weights you can read without one outlier feature dominating.

L1 (Lasso-Style)

The penalty is lambda times the sum of absolute weights. It shrinks aggressively and drives many coefficients to exactly zero, performing built-in feature selection. The result is a sparse model - often the more interpretable one in wide datasets (thousands of candidate features, a handful of real signals). L1 is implemented with subgradient methods because the absolute value is not differentiable at 0.

Elastic Net

Elastic net mixes the two: (1 - ratio)·L1 + ratio·L2. It keeps L1's sparsity while fixing its weakness of picking one feature at random from a correlated group - the L2 part keeps correlated features' coefficients more evenly spread.

The C Parameter in scikit-learn

scikit-learn inverts the convention: C is the inverse of the penalty strength. Small C means strong regularization (more shrinkage), large C means almost none. The default is C=1.0.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(0)
X = rng.normal(size=(300, 12))
signal = X[:, 0] - 1.5 * X[:, 3] + rng.normal(scale=0.4, size=300)
y = (signal > 0).astype(int)

scaler = StandardScaler().fit(X)
for pen, C in [("l1", 0.01), ("l1", 0.1), ("l2", 0.1), ("l2", 100)]:
    m = LogisticRegression(penalty=pen, C=C, solver="liblinear").fit(scaler.transform(X), y)
    nz = int(np.abs(m.coef_[0]) > 1e-6)
    print(pen, f"C={C:<6}", "nonzero features:", nz,
          "max |coef|:", round(float(np.abs(m.coef_[0]).max()), 3))

The L1 rows show few nonzero coefficients (feature selection); the L2 rows keep all twelve but with magnitudes that grow as C grows - shrinkage dialed down.

Choosing

• Default to L2; it is the safer generalization choice and what most production models use.
• Use L1 when you want an automatic shortlist of features or when the dataset is very wide.
• Use elastic net when many features are correlated and you still want sparsity.
• Tune the strength with cross-validation (next chapters), not by feel.

Summary

L2 shrinks all weights smoothly for stability; L1 zeros out useless features for sparsity; elastic net blends the two. In scikit-learn the strength is 1/C - smaller C, more shrinkage - and the intercept is never penalized. Pick L2 by default and let cross-validation set the dial.