Multiclass Logistic Regression

Logistic regression is not limited to two classes. The standard extension is multinomial (softmax) logistic regression: compute one linear score per class and normalize the exponentials into a probability vector. A closely related strategy, one-vs-rest, trains a separate binary model per class. scikit-learn picks between them for you in most cases, and this chapter shows both the math and the API.

The Softmax Function

For class scores z_1, ..., z_K, the softmax is p_k = e^(z_k) / sum_j e^(z_j). It maps any real vector to a probability distribution: all entries positive, all summing to 1, and the entry with the largest score gets the most mass. The exponential rescaling keeps it numerically safe (subtract the max score before exponentiating) and makes probabilities comparable across classes.

import numpy as np

def softmax(z):
    z = z - z.max(axis=1, keepdims=True)   # numerical stability
    e = np.exp(z)
    return e / e.sum(axis=1, keepdims=True)

print(softmax(np.array([[1.0, 3.0, 0.1]])))   # [[0.018 0.968 0.014]]

One-vs-Rest

OvR trains K binary problems: "class k versus everything else," each a plain logistic regression with its own weights. Predictions come from comparing the K probabilities (or, historically, the winning binary decision). OvR is the default when the problem is a set of independent two-class questions, and it is what multi_class="ovr" selects explicitly.

In scikit-learn

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import StandardScaler

data = load_iris()
Xtr, Xte, ytr, yte = train_test_split(data.data, data.target,
                                      test_size=0.3, random_state=0)
scaler = StandardScaler().fit(Xtr)

model = LogisticRegression(max_iter=2000).fit(scaler.transform(Xtr), ytr)
print("accuracy:", round(accuracy_score(yte, model.predict(scaler.transform(Xte))), 3))
print("classes:", model.classes_)
print("coef shape:", model.coef_.shape)   # (3, 4) - one row per class
proba = model.predict_proba(scaler.transform(Xte[:5]))
print("row sums:", np.round(proba.sum(axis=1), 6))   # each row sums to 1

For multiclass with the default lbfgs solver, scikit-learn fits multinomial logistic regression automatically. The coef_ matrix has one row per class; predict_proba returns the full row per sample, so every class probability is available for thresholding or logging.

Reading Multiclass Coefficients

The per-class rows are usually compared against a baseline (often the last class or the mean row): a feature that is positive in class A's row and negative in class B's row is a feature that pushes predictions toward A and away from B. This relative reading is how you explain "why did the model call this a setosa?" - the coefficients show which measurements moved the probabilities.

When OvR Wins

• Very many classes (softmax fits one big model; OvR fits K small ones that can be updated independently per class).
• Asymmetric class pairs where the "everything else" mixture is itself a meaningful negative class.
• Solvers or solvers-with-penalties where the OvR decomposition is cheaper.

Summary

Multinomial logistic regression applies softmax to one linear score per class; one-vs-rest fits K binary models instead. scikit-learn defaults to multinomial for multiclass problems, and coef_ plus predict_proba give you the full per-class picture for both - the right tool whenever the classes are mutually exclusive and the boundary per class is roughly linear.