The Log Loss Cost Function

To learn the weights, the model needs a number that measures how wrong its probabilities are, and that number must be differentiable and, ideally, convex. Logistic regression uses log loss (cross-entropy): it punishes confident wrong predictions much harder than slightly-off right ones, which is exactly the behavior a probability model should have.

Why Not Mean Squared Error

One might try to minimize the squared error between the predicted probability and the label. It works in linear regression, but composed with the sigmoid it becomes a non-convex function of the weights: the loss surface develops flat valleys and multiple local minima, and gradient descent can stall far from the optimum. Log loss composed with the sigmoid is convex in w - there is one global minimum - which is the main mathematical reason logistic regression uses it.

The Formula

For a single sample with label y in {0, 1} and predicted probability p:

loss(y, p) = -[ y * log(p) + (1 - y) * log(1 - p) ]

Read the two cases: if y = 1 the loss is -log(p), which is 0 at p = 1 and grows without bound as p approaches 0. If y = 0 the loss is -log(1 - p), symmetric. The model is trained to minimize the average over all samples.

In Python

import numpy as np

def log_loss(y_true, p):
    eps = 1e-15
    p = np.clip(p, eps, 1 - eps)
    return float(-np.mean(y_true * np.log(p) + (1 - y_true) * np.log(1 - p)))

y = np.array([1, 1, 0, 0, 1, 0])
good = np.array([0.9, 0.8, 0.1, 0.2, 0.95, 0.05])
bad  = np.array([0.2, 0.4, 0.3, 0.6, 0.55, 0.7])

print("calibrated:", round(log_loss(y, good), 4))   # ~0.10
print("confidently wrong:", round(log_loss(y, bad), 4))   # ~0.98

The clipping at 1e-15 avoids log(0); well-behaved models never need it, but it keeps the metric usable mid-training.

The Pleasant Gradient

Log loss was not chosen only for convexity - its gradient with respect to the weights is the cleanest you will meet in machine learning. If p = sigmoid(X·w) and the labels are the matrix y, then:

d(loss)/dw = X^T (p - y) / n

The chain rule cancels the sigmoid's derivative and the cost function's logarithm, leaving "design matrix times residual, divided by n" - the same shape as the linear regression gradient, with (p - y) playing the role of the residual. The next chapter uses this formula to write the update rule.

Summary

Log loss measures the average surprisal of the true labels given the predicted probabilities: confident mistakes hurt a lot, near-certain correct calls cost almost nothing. It is convex with the sigmoid, and its gradient X^T(p - y)/n is the engine of the entire training loop.