PCA (Principal Component Analysis)

PCA is an unsupervised learning technique for dimensionality reduction. PCA transforms data into new coordinates where the first component captures the largest variance.

Basic Concept

• Principal Component: The direction with the largest variance in the data
• Eigenvalues: The amount of variance explained by each component
• Eigenvectors: The direction of a principal component

Why PCA?

• Visualizing high-dimensional data (2D/3D)
• Reducing computational cost
• Reducing noise
• Dealing with the curse of dimensionality

from sklearn.decomposition import PCA
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler

iris = load_iris()
X = StandardScaler().fit_transform(iris.data)

pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)

print(f"Original shape: {iris.data.shape}")
print(f"Reduced shape: {X_pca.shape}")
print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
print(f"Total variance explained: {sum(pca.explained_variance_ratio_):.4f}")

Explained Variance

from sklearn.decomposition import PCA
from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

digits = load_digits()
X = StandardScaler().fit_transform(digits.data)

pca = PCA().fit(X)

plt.figure(figsize=(8, 4))
plt.plot(range(1, len(pca.explained_variance_ratio_) + 1),
         np.cumsum(pca.explained_variance_ratio_), 'bo-')
plt.xlabel('Number of Components')
plt.ylabel('Cumulative Explained Variance')
plt.title('Explained Variance vs Number of Components')
plt.grid(True, alpha=0.3)
plt.axhline(y=0.95, color='r', linestyle='--', label='95%')
plt.legend()
plt.show()

PCA for Visualization

from sklearn.decomposition import PCA
from sklearn.datasets import load_wine
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

wine = load_wine()
X = StandardScaler().fit_transform(wine.data)
y = wine.target

pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)

plt.figure(figsize=(8, 6))
for i, name in enumerate(wine.target_names):
    mask = y == i
    plt.scatter(X_pca[mask, 0], X_pca[mask, 1], label=name, alpha=0.7)

plt.xlabel(f'PC1 ({pca.explained_variance_ratio_[0]:.2%})')
plt.ylabel(f'PC2 ({pca.explained_variance_ratio_[1]:.2%})')
plt.title('PCA - Wine Dataset')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

PCA vs Original Features

from sklearn.decomposition import PCA
from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

digits = load_digits()
X, y = digits.data, digits.target

# Model with all features
pipe_full = make_pipeline(StandardScaler(), SVC())
scores_full = cross_val_score(pipe_full, X, y, cv=5)

# Model with PCA (95% variance)
pipe_pca = make_pipeline(StandardScaler(), PCA(n_components=0.95), SVC())
scores_pca = cross_val_score(pipe_pca, X, y, cv=5)

print(f"Full features ({X.shape[1]}): {scores_full.mean():.4f}")
print(f"PCA features: {scores_pca.mean():.4f}")

PCA Attributes

from sklearn.decomposition import PCA

pca = PCA(n_components=5)
pca.fit(X)

print(f"Components shape: {pca.components_.shape}")
print(f"Explained variance: {pca.explained_variance_}")
print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
print(f"Mean: {pca.mean_}")

Pros and Cons

Tips

• Scaling is mandatory before PCA
• Choose n_components based on cumulative explained variance ≥ 95%
• Use Kernel PCA for non-linear data

Next chapter: Feature Engineering, the art of creating new, more informative features.

Next: Feature Engineering →