Logistic Regression

Logistic Regression, despite having "regression" in its name, is actually a classification algorithm used to predict the probability of an event. This algorithm is very popular for binary classification.

Sigmoid Function

Logistic Regression uses the sigmoid function to transform a linear output into a probability between 0 and 1:

σ(z) = 1 / (1 + e^(-z))

Where z = b0 + b1*x1 + ... + bn*xn

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

z = np.linspace(-10, 10, 100)
plt.figure(figsize=(8, 4))
plt.plot(z, sigmoid(z))
plt.axhline(y=0.5, color='r', linestyle='--', alpha=0.5)
plt.xlabel('z')
plt.ylabel('sigmoid(z)')
plt.title('Sigmoid Function')
plt.grid(True, alpha=0.3)
plt.show()

Decision Boundary

The decision boundary is the line/divider that separates the classes. Default threshold 0.5:
- If σ(z) ≥ 0.5 → Class 1 (positive)
- If σ(z) < 0.5 → Class 0 (negative)

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=200, n_features=2, n_redundant=0,
                           n_informative=2, random_state=42, n_clusters_per_class=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

model = LogisticRegression()
model.fit(X_train, y_train)

accuracy = model.score(X_test, y_test)
print(f"Accuracy: {accuracy:.4f}")

Multi-Class Classification

For classification with more than 2 classes, Logistic Regression uses the following strategies:

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression

iris = load_iris()
X, y = iris.data, iris.target

model = LogisticRegression(max_iter=200)
model.fit(X, y)

score = model.score(X, y)
print(f"Accuracy Iris dataset: {score:.4f}")
print(f"Number of classes: {len(model.classes_)}")

Evaluation

Logistic Regression with Regularization

from sklearn.linear_model import LogisticRegression

# L1 Regularization (Lasso)
model_l1 = LogisticRegression(penalty='l1', solver='liblinear', C=1.0)

# L2 Regularization (Ridge) - default
model_l2 = LogisticRegression(penalty='l2', C=1.0)

# Without regularization
model_none = LogisticRegression(penalty=None)

Example: Spam Email Detection

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

emails = [
    "Congratulations, you have won a cash prize worth millions",
    "Meeting at 3 PM in the main room",
    "Click this link to claim your bonus",
    "The monthly report is finished",
    "90% discount promo today only",
    "Reminder: project deadline tomorrow"
]
labels = [1, 0, 1, 0, 1, 0]  # 1=spam, 0=ham

pipeline = make_pipeline(TfidfVectorizer(), LogisticRegression())
pipeline.fit(emails, labels)

prediction = pipeline.predict(["You won the raffle! Click now"])
print(f"Prediction: {'Spam' if prediction[0] == 1 else 'Ham'}")

When to Use Logistic Regression?

• Binary classification with linear separability
• When model interpretability matters
• First baseline for a classification problem
• Datasets that are not too large

The next chapter covers a distance-based algorithm: K-Nearest Neighbors.

Next: K-Nearest Neighbors →