Gradient Descent for Logistic Regression

With a convex cost function and a clean gradient, training is gradient descent: start from some weights, repeatedly move a small step in the direction that reduces the loss, and stop when the steps get tiny. Because the loss is convex, any local minimum is the global minimum, so the method is guaranteed to find the best weights (for a given regularization) - the only question is how fast.

The Update Rule

Let Xb be the design matrix with a column of ones prepended (so b is just another weight), p = sigmoid(Xb·w) the predicted probabilities, and y the labels. One batch (full-batch) step is:

w = w - lr * Xb^T (p - y) / n

Intuition per weight: the residual (p - y) is positive where the model overshot and negative where it undershot; multiplying by Xb attributes that error back to each feature, and the step nudges every weight away from its mistakes. A large learning rate oscillates or diverges; a tiny one crawls. Typical starting values for scaled data are 0.01 to 1.0.

A Mini Implementation

import numpy as np

rng = np.random.default_rng(1)
X = rng.normal(size=(100, 2))
X[:50] += np.array([1.5, 1.5])
y = np.array([0] * 50 + [1] * 50)
Xb = np.c_[np.ones(len(X)), X]

w = np.zeros(3)
lr, epochs = 0.5, 300
for epoch in range(epochs):
    p = 1.0 / (1.0 + np.exp(-Xb @ w))
    grad = Xb.T @ (p - y) / len(y)
    w -= lr * grad

print("weights:", np.round(w, 3))   # sign and scale match sklearn's fit

Run the same data through sklearn.linear_model.LogisticRegression and the weight directions will agree (up to the intercept convention and the solver's tolerance) - a good sanity check that your math is right.

Batch, Mini-Batch, Stochastic

• Full batch: one step per epoch, exact gradient. Fine up to a few hundred thousand rows; the matrix product gets expensive beyond that.
• Mini-batch (the default in practice): 32-512 samples per step. Approximate gradient, vectorized, 10-100x faster per epoch than full batch on large data. scikit-learn's saga and sag solvers are mini-batch.
• Stochastic (batch size 1): noisest but cheapest step; rarely used directly for this model.

Convergence and Stopping

Stop when the change in weights or the change in loss falls below a tolerance (scikit-learn's tol=1e-4 default) or when the iteration cap is hit. With a learning rate that decays (or a solver that uses line search), the trajectory approaches the optimum and settles. If the loss bounces or climbs, the learning rate is too high; if it crawls linearly for thousands of epochs, it is too low.

Summary

Logistic regression trains by iterating w = w - lr·X^T(p - y)/n until the residuals stop changing. The convexity of log loss guarantees convergence to the global optimum, mini-batch steps make it scalable, and the learning rate is the one knob that decides whether training is fast, stable, or both.