Data Preprocessing for AI

Machine learning models are only as good as the data they consume. Real-world datasets arrive incomplete, inconsistent, and on wildly different scales. Data preprocessing converts that raw material into a clean, numeric, well-shaped matrix that scikit-learn or a deep learning framework can learn from. In practice this stage consumes most of the time in an AI project, so mastering it with pandas and NumPy pays off for every model you build later.

Handling Missing Values

The first inspection task is finding gaps. pandas exposes missing data as NaN, and df.isnull().sum() gives a per-column count. Small gaps in important columns are usually filled rather than dropping whole rows, which would discard valuable signal.

import pandas as pd

df = pd.read_csv("customers.csv")
print(df.isnull().sum())

# Impute numeric columns with the median, robust to outliers
df["income"] = df["income"].fillna(df["income"].median())

# Impute categorical columns with the most frequent value
df["city"] = df["city"].fillna(df["city"].mode()[0])

For pipelines that must apply the same rule at prediction time, prefer the sklearn imputers, because they store the fitted statistics and reuse them consistently:

from sklearn.impute import SimpleImputer

num_imputer = SimpleImputer(strategy="median")
df[["age", "income"]] = num_imputer.fit_transform(df[["age", "income"]])

Encoding Categorical Features

Most estimators expect numbers, not strings. Ordinal categories (small, medium, large) can be mapped to integers; nominal categories (city names) should use one-hot encoding so the model does not invent a false ordering.

import pandas as pd

# One-hot encode nominal columns
df = pd.get_dummies(df, columns=["city", "device"], drop_first=True)

# Ordinal encode a natural ranking
size_map = {"small": 0, "medium": 1, "large": 2}
df["size_code"] = df["size"].map(size_map)

Feature Scaling

Distance-based methods such as k-NN, SVM, k-means, and any gradient-descent model converge poorly when one feature spans 0–100,000 while another spans 0–1. Standardization (zero mean, unit variance) is the default choice; MinMax scaling maps values into a fixed range and suits neural networks. Fit the scaler on training data only, then transform validation and test data with the same parameters.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X = df.drop("churned", axis=1)
y = df["churned"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)  # never fit on test data

Feature Engineering

Preprocessing is also where you create new signal. Dates decompose into day of week and month, ratios combine totals with counts, and lag features summarize history. A sklearn Pipeline chains imputation, encoding, and scaling so the entire transformation is one fitted object that can be saved and reused in production.

from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
    ("model", LogisticRegression(max_iter=1000)),
])
pipe.fit(X_train, y_train)

Summary

Data preprocessing means finding and fixing missing values, converting categorical text into useful numbers, scaling features onto comparable ranges, and engineering new indicators — all wrapped in reproducible pipelines. Cleaning the data before modeling removes an entire class of bugs later, and a fitted pipeline guarantees that training-time and serving-time features behave identically.