Skillversitas

Polynomial Regression

Polynomial regression extends linear regression to curved relationships: instead of predicting a straight line, the model fits a polynomial - a curve - to the data. The trick that makes it "still linear regression" is feature expansion: generate the powers of the input as new features, and the model fits a line in that expanded space. The curve's degree is the one hyperparameter that matters, and it is a pure bias-variance dial: low degree underfits, high degree memorizes the noise.

The Idea in One Picture

With one input feature x, expanding to degree d creates the features [1, x, x^2, ..., x^d]. A linear model on those features produces a degree-d polynomial in x. So "polynomial regression" is linear regression with engineered features - and the same tools (coefficients, R2, residuals, regularization) apply unchanged.

Fitting with PolynomialFeatures

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(1)
x = np.sort(rng.uniform(-3, 3, 100)).reshape(-1, 1)
y = 0.5 * x ** 2 - x + rng.normal(scale=0.5, size=100)

pipe = make_pipeline(
    PolynomialFeatures(degree=3, include_bias=False),
    LinearRegression(),
)
pipe.fit(x, y)

x_grid = np.linspace(-3, 3, 200).reshape(-1, 1)
plt.scatter(x, y, alpha=0.5, label="data")
plt.plot(x_grid, pipe.predict(x_grid), label="degree 3 fit")
plt.legend()
plt.show()

PolynomialFeatures(degree=d) expands the input into all monomials up to degree d - for multiple features, that includes the interactions (x1*x2, x1^2, ...) up to the degree, which is why the feature count grows fast. Wrapping it in a pipeline with LinearRegression keeps the expansion and the fit as one object, so fit, predict, and cross-validation all work on the pipeline.

The Degree Trade-Off

import numpy as np
from sklearn.model_selection import cross_val_score

scores = {}
for d in range(1, 9):
    p = make_pipeline(PolynomialFeatures(degree=d, include_bias=False),
                      LinearRegression())
    s = cross_val_score(p, x, y, cv=5, scoring="r2")
    scores[d] = (s.mean(), s.std())
    print(f"degree {d}: CV R2 = {s.mean():.3f} +/- {s.std():.3f}")

The pattern is textbook: R2 climbs as the degree rises, peaks where the polynomial matches the true shape, and then degrades - the high-degree fit swings wildly between the training points (variance) and the cross-validation score falls. The peak of the CV curve is the degree to ship. With a little noise in the data, the peak is a small region, and any degree inside it is defensible; the degree far past it is overfitting with a visible wiggly fit.

Regularizing the Expansion

High-degree polynomials have many features, and many features invite overfitting. The fix is the same regularized regression from the linear chapter:

from sklearn.linear_model import Ridge

ridge_pipe = make_pipeline(
    PolynomialFeatures(degree=6, include_bias=False),
    Ridge(alpha=1.0),     # the penalty tames the high-degree coefficients
)
ridge_pipe.fit(x, y)
print(ridge_pipe.predict(x_grid[:3]).round(3))

Ridge (or Lasso, which zeros out the highest-degree terms) lets you use a higher expansion degree for flexibility while the penalty keeps the wild coefficients in check. Tuning alpha plus degree together is a small two-dimensional search - a perfect job for the grid-search chapter later.

When to Use (and Not)

  • Use it when the feature-target relationship is clearly curved and smooth (physics-like responses, diminishing returns) and there is one or two features to expand - the interpretable middle ground between a line and a black box.
  • Do not use it with many features (the monomial explosion makes the design matrix huge), with non-smooth relationships (a tree handles step changes better), or when the curve's shape is not a polynomial at all (try splines or a kernel method).

Summary

Polynomial regression is linear regression on expanded features: PolynomialFeatures(degree=d) plus a linear model in a pipeline. Choose d by cross-validation - the R2 peak is the answer - and tame high degrees with Ridge/Lasso regularization. It is the right tool for smooth, curved, low-dimensional relationships and the wrong one for high-dimensional or jagged ones, where the expansion cost and the shape mismatch both work against it.