Logistic Regression from Scratch

This chapter assembles everything so far - the sigmoid, the log-loss gradient, the gradient descent loop - into a working model class in pure NumPy. It is about sixty lines, has no dependencies beyond NumPy, and matches scikit-learn's output on the same data, which makes it the best way to verify that you actually understand the algorithm.

The Model Class

import numpy as np

class LogisticRegressionScratch:
    def __init__(self, lr=0.1, epochs=1000, l2=0.0):
        self.lr, self.epochs, self.l2 = lr, epochs, l2

    def fit(self, X, y):
        n, d = X.shape
        Xb = np.c_[np.ones(n), X]          # intercept as a column of ones
        w = np.zeros(d + 1)
        for _ in range(self.epochs):
            p = 1.0 / (1.0 + np.exp(-Xb @ w))
            grad = Xb.T @ (p - y) / n
            grad[1:] += self.l2 * w[1:]    # L2 shrinks weights, not the intercept
            w -= self.lr * grad
        self.w = w
        return self

    def decision_function(self, X):
        return np.c_[np.ones(len(X)), X] @ self.w

    def predict_proba(self, X):
        p1 = 1.0 / (1.0 + np.exp(-self.decision_function(X)))
        return np.c_[1 - p1, p1]

    def predict(self, X):
        return (self.predict_proba(X)[:, 1] >= 0.5).astype(int)

Running It

import numpy as np

rng = np.random.default_rng(42)
X = rng.normal(size=(500, 4))
true_w = np.array([1.5, -2.0, 0.8, 0.0])
z = X @ true_w + rng.normal(scale=0.5, size=500)
y = (1 / (1 + np.exp(-z)) > rng.random(500)).astype(int)

model = LogisticRegressionScratch(lr=0.5, epochs=500).fit(X, y)
print("learned:", np.round(model.w, 3))
print("target :", np.r_[0, true_w])   # intercept 0 by construction

The learned coefficients land near the true values up to a common scale factor - logistic regression identifies the weight direction exactly but the overall magnitude is set by how far the data is from the boundary (noisy data needs gentler weights).

Scale Matters

Gradient descent with a fixed learning rate assumes comparable feature scales. If one feature ranges 0-1 and another 0-1,000,000, the loss surface is a narrow valley and a single learning rate that is safe for the big feature is glacially slow for the small one. Standardize features (zero mean, unit variance) before training - the same requirement scikit-learn has, and the reason the next chapters always show a StandardScaler.

What to Extend

• Add early stopping: track the loss each epoch and stop when its improvement is below a tolerance.
• Swap full-batch for mini-batch steps (sample 64 rows per iteration) to scale to large data.
• Add L1 with a subgradient update to get feature selection.
• Wrap it in a small grid search over learning rate and penalty.

Summary

A from-scratch logistic regression is a loop: predict with the sigmoid, compute X^T(p - y)/n, add the penalty, step the weights. Match it against scikit-learn on the same data, standardize your features, and you have a model you can debug line by line - a far more reliable foundation than a black box.