Skillversitas

Feature Engineering

Feature engineering is the art of turning raw columns into inputs a model can use well: encoding categories, scaling magnitudes, extracting time information, and building derived variables that capture the relationships the raw columns hide. It is the step where domain knowledge pays off most - a well-chosen interaction or a log transform routinely beats adding model complexity, and it is where the gap between a naive pipeline and a strong one usually opens.

Encoding Categoricals

import pandas as pd
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder

df["region"] = pd.Categorical(df["region"])

# one-hot: one column per category - the default for trees and linear models
ohe = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
encoded = ohe.fit_transform(df[["region", "plan"]])
print(encoded[:5])

# ordinal: only when the categories have a real order
ord_enc = OrdinalEncoder(unknown_value_encoded=0)
ord_enc.fit(df[["seniority"]])   # e.g. junior < mid < senior

One-hot is the safe default: it makes no assumption about distances between categories and works with every model. Ordinal encoding is correct only for genuinely ordered categories (seniority, size class). handle_unknown="ignore" is mandatory for production: a category seen only in training becomes all-zeros at prediction time instead of an exception.

Scaling Numeric Features

from sklearn.preprocessing import StandardScaler, RobustScaler

scaler = StandardScaler().fit(df[["amount", "age"]])   # fit on TRAIN only
scaled = scaler.transform(df[["amount", "age"]])
# every feature now has mean 0, variance 1
# RobustScaler: median/IQR - use when outliers distort the mean

Scaling is required by distance-based models (KNN, SVM, linear regression with regularization) and helpful for gradient-based training; trees do not need it but it is harmless. The discipline from the cleaning chapter applies: fit on training data, transform everything.

Time Features

df["ts"] = pd.to_datetime(df["created_at"])
df["hour"] = df["ts"].dt.hour
df["day_of_week"] = df["ts"].dt.dayofweek
df["is_weekend"] = (df["day_of_week"] >= 5).astype(int)
df["month"] = df["ts"].dt.month
df["days_since_registration"] = (df["ts"] - df["registered_at"]).dt.days

Raw timestamps are useless to most models; the extracted components are the signal. Hour and day-of-week capture daily and weekly cycles; durations (days since, days to) capture elapsed time; is_weekend captures the work/non-work split. For seasonal data, sinusoidal encodings of the cyclic variables (sin/cos of the angle) keep December adjacent to January.

Derived and Interaction Features

df["price_per_unit"] = df["total"] / df["quantity"].replace(0, pd.NA)
df["log_amount"] = (df["amount"] + 1).apply(lambda v: __import__("math").log1p(v))
df["amount_x_region_us"] = df["amount"] * (df["region"] == "us").astype(int)
df["amount_bin"] = pd.qcut(df["amount"], 4, labels=False, duplicates="drop")

The high-value moves: ratios that normalize one quantity by another (price per unit, response rate), log transforms for right-skewed magnitudes, interactions that say "this only matters when that," and binning that turns a continuous variable into regimes. Each derived feature is a hypothesis - "this relationship exists and helps" - and the evaluation step is what confirms it.

Rules of the Road

  • Every transformation is fitted on training data only - encoders, scalers, bin edges, all of it.
  • Track the feature list: a model with 40 engineered columns needs a registry of what each one means, or nobody can audit it.
  • Ablate: drop the weakest engineered features and check the score - dead weight is maintenance cost.
  • Leakage check: no feature may use information that would not exist at prediction time.

Summary

Engineer features in four families: one-hot for categories (ordinal only when truly ordered), scaling for magnitudes, time components for timestamps, and derived variables for the relationships the raw columns hide. Fit every transformer on training data, keep a registry of what each feature means, and let the evaluation step - not the intuition - decide which features earn their place in the final model.