Feature Selection

Feature Selection is the process of choosing the best subset of features from your data to improve model performance, reduce overfitting, and speed up training.

Filter Methods

Filter methods evaluate features based on statistics without involving a model.

Correlation

import pandas as pd
import numpy as np

np.random.seed(42)
df = pd.DataFrame({
    'feature1': np.random.randn(100),
    'feature2': np.random.randn(100),
    'feature3': np.random.randn(100),
    'target': np.random.randn(100)
})

corr = df.corr()
print(corr['target'].sort_values(ascending=False))

Chi-Square

For categorical features vs a categorical target:

from sklearn.feature_selection import SelectKBest, chi2
from sklearn.datasets import load_iris

iris = load_iris()
selector = SelectKBest(chi2, k=2)
X_selected = selector.fit_transform(iris.data, iris.target)

print(f"Original features: {iris.data.shape[1]}")
print(f"Selected features: {X_selected.shape[1]}")
print(f"Scores: {selector.scores_}")

Mutual Information

from sklearn.feature_selection import mutual_info_classif, mutual_info_regression
from sklearn.datasets import load_iris

iris = load_iris()
mi_scores = mutual_info_classif(iris.data, iris.target, random_state=42)

for name, score in zip(iris.feature_names, mi_scores):
    print(f"{name:20s}: {score:.4f}")

Wrapper Methods

Wrapper methods use a model to evaluate subsets of features.

Forward Selection

from sklearn.feature_selection import SequentialFeatureSelector
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris

iris = load_iris()
model = RandomForestClassifier(n_estimators=100, random_state=42)

sfs = SequentialFeatureSelector(model, n_features_to_select=2,
                                 direction='forward', cv=5)
sfs.fit(iris.data, iris.target)

selected = sfs.get_support()
print(f"Selected features: {[iris.feature_names[i] for i, s in enumerate(selected) if s]}")

Backward Elimination

sfs_backward = SequentialFeatureSelector(model, n_features_to_select=2,
                                          direction='backward', cv=5)
sfs_backward.fit(iris.data, iris.target)

selected = sfs_backward.get_support()
print(f"Selected features: {[iris.feature_names[i] for i, s in enumerate(selected) if s]}")

Embedded Methods

Embedded methods perform selection during model training.

Lasso (L1 Regularization)

from sklearn.linear_model import LassoCV
from sklearn.datasets import make_regression
import numpy as np

X, y = make_regression(n_samples=100, n_features=20, n_informative=5, random_state=42)

lasso = LassoCV(cv=5, random_state=42)
lasso.fit(X, y)

important_features = np.where(lasso.coef_ != 0)[0]
print(f"Selected features: {important_features}")
print(f"Number of selected features: {len(important_features)}")

Tree-Based Feature Importance

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
import pandas as pd

iris = load_iris()
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(iris.data, iris.target)

importances = pd.Series(model.feature_importances_, index=iris.feature_names)
importances = importances.sort_values(ascending=False)

print("Feature Importances:")
print(importances)

Method Comparison

RFE (Recursive Feature Elimination)

from sklearn.feature_selection import RFE
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_wine

wine = load_wine()
model = RandomForestClassifier(n_estimators=100, random_state=42)

rfe = RFE(model, n_features_to_select=5, step=1)
rfe.fit(wine.data, wine.target)

selected = [wine.feature_names[i] for i, s in enumerate(rfe.support_) if s]
print(f"Top 5 features: {selected}")

Feature Selection Tips

• Start with filter methods for initial exploration
• Use embedded methods (tree importance) for a baseline
• Use wrapper methods for final fine-tuning
• Watch out for multicollinearity between features
• Always validate with cross-validation

Next chapter: Data Preprocessing, an essential step before modeling.

Next: Data Preprocessing →