Skillversitas

PCA and Dimensionality Reduction

Principal component analysis (PCA) compresses a feature matrix into fewer dimensions by projecting it onto the directions of maximum variance. The first principal component is the line through the data along which it spreads most; the second is the next-most-variance direction, orthogonal to the first; and so on. The result is a low-dimensional representation that keeps most of the information, which serves three purposes: visualization of high-dimensional data, noise reduction, and speedup of downstream models that choke on wide, correlated inputs.

Fitting and Transforming

import numpy as np
from sklearn.decomposition import PCA
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler

data = load_iris(as_frame=True)
X = data.frame.drop(columns="target")
scaler = StandardScaler().fit(X)
Xs = scaler.transform(X)

pca = PCA(n_components=0.95)      # keep enough components for 95% of variance
pca.fit(Xs)
print("components kept:", pca.n_components_)
print("explained variance ratio:", np.round(pca.explained_variance_ratio_, 3))
print("cumulative:", np.round(np.cumsum(pca.explained_variance_ratio_), 3))

Xr = pca.transform(Xs)            # the compressed data
print("shape:", Xs.shape, "->", Xr.shape)

Three things to read from a fitted PCA. explained_variance_ratio_ says what fraction of the total variance each component holds - the curve of cumulative ratios is the compression menu (95% in two components is a strong result). n_components_ is how many were kept under the 0.95 rule. And the transformed matrix is the data in the new basis, ready for plotting or modeling. Scaling first is standard practice: PCA finds variance directions, and raw-scale differences would make the largest-magnitude feature dominate the first component regardless of its informational content.

Visualizing High-Dimensional Data

import matplotlib.pyplot as plt

pca2 = PCA(n_components=2).fit(Xs)
X2 = pca2.transform(Xs)
y = data.frame["target"]

plt.scatter(X2[:, 0], X2[:, 1], c=y, cmap="viridis", alpha=0.7)
plt.xlabel(f"PC1 ({pca2.explained_variance_ratio_[0]:.1%} var)")
plt.ylabel(f"PC2 ({pca2.explained_variance_ratio_[1]:.1%} var)")
plt.show()

The two-component projection is the standard "show me the structure" plot: classes that separate in PC space were already nearly separable in the original space (the linear structure PCA captured), and classes that overlap in the projection may still be separable by non-linear or higher-order combinations. The axis labels must carry the variance fractions - a PCA plot with unlabeled axes is a picture, not a measurement.

What the Components Mean

pca.components_ holds the loading vectors - each row is a component expressed as a weighted combination of the original features. A component whose loadings are all positive and roughly equal is "overall size"; one with a large positive loading on one feature and negative on another is "that feature versus that one." Reading the loadings is how you name the components in a report ("PC1 is the activity axis") and how you decide whether the compression threw away anything meaningful - if a business-critical feature loads heavily on a discarded component, keep more components.

When PCA Is (and Is Not) the Tool

  • Use it for visualization of >3-D data, for removing near-duplicate correlated features before a linear model, for denoising (keeping the top components discards the low-variance, high-noise directions), and for compressing wide inputs before a slow downstream model.
  • Do not use it as a feature-reduction shortcut when the model does its own selection (trees handle redundant features fine), when the variance structure is not the signal structure (variance can be dominated by noise or by one noisy feature), or when interpretability of the reduced features matters more than their count.
  • Non-linear alternatives: t-SNE and UMAP for visualization-only projections of very high-dimensional data (they preserve local structure better but are not linear, not invertible, and not a preprocessing step for modeling).

Summary

Apply PCA to scaled data with a variance threshold (0.95 is a reasonable default), read the cumulative explained-variance curve to choose the component count, and interpret the result through components_ - the loadings tell you what each axis actually measures. Use the two-component projection to see cluster and class structure, keep the variance fractions on the axis labels, and treat PCA as a compression-and-inspection tool rather than a free accuracy booster: it helps most when the original features are redundant and the model is linear.