Linear Regression

Linear Regression is the most basic supervised learning algorithm, used to predict a continuous value based on the linear relationship between independent and dependent variables.

Basic Concept

The simple linear regression equation:

y = mx + b

Where:
- y: dependent variable (target/prediction)
- x: independent variable (feature)
- m: slope/gradient
- b: intercept

For multiple linear regression:

y = b0 + b1*x1 + b2*x2 + ... + bn*xn

Cost Function (MSE)

To evaluate how well the model performs, we use Mean Squared Error:

import numpy as np

def mse(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

MSE measures the mean of the squared differences between the actual values and the predictions.

Gradient Descent

Gradient descent is an optimization algorithm for finding the best parameter values (m and b). Its basic principle:

1. Initialize the parameters randomly
2. Compute the gradient (slope) of the cost function
3. Update the parameters in the direction of the negative gradient
4. Repeat until convergence

def gradient_descent(X, y, learning_rate=0.01, iterations=1000):
    m = 0
    b = 0
    n = len(X)

    for _ in range(iterations):
        y_pred = m * X + b
        dm = (-2/n) * np.sum(X * (y - y_pred))
        db = (-2/n) * np.sum(y - y_pred)
        m -= learning_rate * dm
        b -= learning_rate * db

    return m, b

Implementation with Scikit-Learn

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np

np.random.seed(42)
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

model = LinearRegression()
model.fit(X_train, y_train)

print(f"Intercept: {model.intercept_[0]:.2f}")
print(f"Coefficient: {model.coef_[0][0]:.2f}")

y_pred = model.predict(X_test)
print(f"MSE: {mean_squared_error(y_test, y_pred):.4f}")
print(f"R² Score: {r2_score(y_test, y_pred):.4f}")

Model Evaluation

Case Study: House Price Prediction

import pandas as pd
from sklearn.linear_model import LinearRegression

data = {
    'area': [50, 60, 70, 80, 90, 100, 110, 120],
    'price': [350, 400, 450, 520, 570, 650, 700, 800]
}
df = pd.DataFrame(data)

X = df[['area']]
y = df['price']

model = LinearRegression()
model.fit(X, y)

new_prediction = model.predict([[130]])
print(f"Predicted price for 130 m²: $ {new_prediction[0]:,.0f} million")

Limitations

• Only captures linear relationships
• Very sensitive to outliers
• Assumption: residuals are normally distributed
• Multicollinearity can distort the model

For classification problems (category labels), we need a different approach. The next chapter covers Logistic Regression.

Next: Logistic Regression →