Feature Engineering

Feature Engineering is the process of creating new features from raw data to improve ML model performance. Good features can have a bigger impact than the choice of algorithm.

Polynomial Features

Creating interaction and power features from existing features:

from sklearn.preprocessing import PolynomialFeatures
import numpy as np

X = np.array([[2, 3], [4, 5]])
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)

print(f"Original: {X.shape}")
print(f"Polynomial: {X_poly.shape}")
print(f"Feature names: {poly.get_feature_names_out()}")

Binning (Discretization)

Converting a continuous feature into discrete categories:

import pandas as pd
import numpy as np

np.random.seed(42)
ages = np.random.randint(18, 70, 100)

df = pd.DataFrame({'age': ages})

# Equal-width binning
df['age_group'] = pd.cut(df['age'], bins=5, labels=['Young', 'Adult', 'Middle', 'Senior', 'Elderly'])

# Custom binning
bins = [0, 25, 35, 50, 100]
labels = ['Young', 'Adult', 'Middle', 'Senior']
df['age_category'] = pd.cut(df['age'], bins=bins, labels=labels)

print(df[['age', 'age_group', 'age_category']].head(10))

Log Transform

Handling a right-skewed distribution:

import numpy as np
import pandas as pd

np.random.seed(42)
income = np.random.lognormal(mean=10, sigma=1, size=1000)
df = pd.DataFrame({'income': income})

df['log_income'] = np.log1p(df['income'])

print(f"Original skew: {df['income'].skew():.4f}")
print(f"Log skew: {df['log_income'].skew():.4f}")

Categorical Encoding

One-Hot Encoding

import pandas as pd
from sklearn.preprocessing import OneHotEncoder

df = pd.DataFrame({
    'color': ['red', 'blue', 'green', 'red', 'blue'],
    'size': ['S', 'M', 'L', 'XL', 'M']
})

encoder = OneHotEncoder(sparse_output=False, drop='first')
encoded = encoder.fit_transform(df[['color']])
print(encoded)

Label Encoding

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()
df['color_encoded'] = le.fit_transform(df['color'])
print(df[['color', 'color_encoded']])

Target Encoding

import pandas as pd
import numpy as np

np.random.seed(42)
df = pd.DataFrame({
    'city': np.random.choice(['New York', 'Chicago', 'Houston'], 100),
    'price': np.random.randint(100, 1000, 100)
})

target_mean = df.groupby('city')['price'].mean()
df['city_encoded'] = df['city'].map(target_mean)
print(df.head(10))

Feature Creation from Dates

import pandas as pd
import numpy as np

np.random.seed(42)
dates = pd.date_range('2024-01-01', periods=100, freq='D')
df = pd.DataFrame({'date': dates, 'sales': np.random.randint(100, 1000, 100)})

df['day_of_week'] = df['date'].dt.dayofweek
df['month'] = df['date'].dt.month
df['is_weekend'] = df['day_of_week'].isin([5, 6]).astype(int)
df['quarter'] = df['date'].dt.quarter

print(df[['date', 'day_of_week', 'month', 'is_weekend', 'quarter']].head())

Scaling and Normalization

from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler
import numpy as np

X = np.array([[100, 0.001], [200, 0.002], [150, 0.0015], [300, 0.003]])

standard = StandardScaler()
minmax = MinMaxScaler()
robust = RobustScaler()

print("StandardScaler:")
print(standard.fit_transform(X))
print("\nMinMaxScaler:")
print(minmax.fit_transform(X))

Feature Engineering Tips

Next chapter: Feature Selection, how to choose the best features for your model.

Next: Feature Selection →