Skillversitas

Linear Regression

Linear regression is the simplest predictive model and the right first one for any regression problem: it predicts the target as a weighted sum of the features plus an intercept, and it learns the weights by minimizing the squared error. It is fast, interpretable (every coefficient is a direct statement about a feature's effect), and its assumptions are explicit enough to check - which makes it the baseline that every fancier regression model must beat.

Fitting in scikit-learn

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_absolute_error, mean_squared_error

rng = np.random.default_rng(0)
X = rng.normal(size=(500, 3))
y = 4 * X[:, 0] - 2 * X[:, 1] + 0.5 * X[:, 2] + rng.normal(scale=1.0, size=500)

Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)

model = LinearRegression().fit(Xtr, ytr)
print("intercept:", round(model.intercept_, 3))
print("coef:", np.round(model.coef_, 3))

pred = model.predict(Xte)
print("R2:", round(r2_score(yte, pred), 4))
print("MAE:", round(mean_absolute_error(yte, pred), 4))
print("RMSE:", round(mean_squared_error(yte, pred, squared=False), 4))

The coefficients land near the true weights (4, -2, 0.5) because the data was generated linearly - when the relationship is linear, linear regression recovers it almost exactly. R2 is the fraction of target variance explained (1.0 = perfect, 0.0 = no better than predicting the mean), MAE is the average absolute error in target units, and RMSE is like MAE but punishes large errors harder.

Reading the Coefficients

A coefficient of 4 on feature 0 means: one unit more of feature 0 raises the prediction by 4, holding everything else fixed. On standardized features the coefficients become directly comparable - the feature with the largest |coefficient| is the most influential. That comparability is a real reason to standardize before interpreting, and the coefficients plus their order of magnitude are the model's entire explanation.

The Assumptions to Check

  • Linearity: plot each feature against the target (or the residuals against the predictions). A curve in that plot means the linear form is wrong - polynomial or a tree-based model.
  • Homoscedasticity: the residual spread should be constant across prediction values. Fanning out means errors grow with the target - a log transform or weighted regression helps.
  • No multicollinearity: two features that are nearly duplicates make individual coefficients unstable (large, opposite-signed, swapping between refits). The prediction stays fine; the interpretation does not.
  • Independent errors: in time-ordered data, correlated residuals (trends, cycles) violate the assumptions - a time-series-aware model is needed.
import matplotlib.pyplot as plt

resid = yte - pred
fig, (a1, a2) = plt.subplots(1, 2)
a1.scatter(pred, resid)
a1.axhline(0, color="gray")
a1.set_xlabel("predicted"); a1.set_ylabel("residual")
a1.scatter(yte, resid)
a2.set_xlabel("actual"); a2.set_ylabel("residual")
fig.tight_layout()
plt.show()

Regularized Variants

When there are many features or correlations between them, the plain model overfits. The regularized variants add a penalty on the weights: Ridge (L2, shrinks all weights), Lasso (L1, zeros out the useless ones - built-in feature selection), and ElasticNet (both). The API is identical - swap the class, tune alpha - and the choice of which one to use is a hyperparameter decision covered in the tuning chapter.

Summary

Fit LinearRegression on any regression problem as the baseline, read the coefficients as per-unit effects (on standardized features for comparability), and check the four assumptions with residual plots. Report R2 plus MAE/RMSE in target units, reach for Ridge/Lasso/ElasticNet when the feature set is wide, and remember that a good R2 on a linear dataset is the model working as designed - not a fluke to be explained away.