Exploratory Data Analysis

Exploratory data analysis (EDA) is the step where you meet the data before the model does: what each column contains, what is missing, where the outliers sit, and which variables move together. It takes a fraction of the project's time and prevents most of its failures - a model trained on a column that is 80% missing, or a target that is constant within each class, will fail in ways that are embarrassing to discover late. EDA is mostly one-line pandas calls plus a few plots, run with the question "what would surprise me?" in mind.

The Numeric Summary

import pandas as pd

df = pd.read_csv("orders.csv")
print(df.describe())               # count, mean, std, min, 25/50/75%, max
print(df.describe(include="all"))  # include categorical columns too
print(df.isna().sum().sort_values(ascending=False))   # missing per column

describe gives every numeric column's distribution in one table - the quartiles reveal skew (median far from mean), the min/max reveal impossible values (negative ages, 1900 dates in a 2026 dataset). The missing-value count is the single most informative first number: a column that is half empty is either a feature to impute, a feature to drop, or a signal in itself ("missing" as a category).

Distributions and Relationships

import matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 3, figsize=(12, 3.5))
df["amount"].hist(bins=40, ax=axes[0])
df["amount"].boxplot(ax=axes[1])
df.plot.scatter("features_x", "amount", ax=axes[2], alpha=0.4)
fig.tight_layout()
plt.show()

print(df.corr().round(3))          # the correlation matrix

Three views cover most numeric EDA: a histogram for shape (skew, multimodality, gaps), a box plot for spread and outliers, and a scatter against the target for signal. The correlation matrix then quantifies linear relationships - high correlation between features is a warning (redundancy, or a leakage candidate), and correlation with the target is the signal you are hunting for.

Categorical Columns

print(df["region"].value_counts())
print(df["region"].value_counts(normalize=True))   # shares, not counts

# the target rate per category - the classic EDA table
print(df.groupby("region")["churned"].agg(["count", "mean"]).round(3))

For categorical columns, value_counts shows the distribution and groupby-plus-target gives the strongest early signal in the whole dataset: if the churn rate is 2% in one region and 22% in another, that column alone is a model. Low-cardinality columns (under ~50 values) also get a cross-tabulation against the target to catch the obvious winners before any modeling.

Outliers and Duplicates

# how many rows are exact duplicates?
print(df.duplicated().sum())

# the extreme tail of a numeric column
q1, q3 = df["amount"].quantile([0.25, 0.75])
iqr = q3 - q1
outliers = df[(df["amount"] < q1 - 1.5 * iqr) | (df["amount"] > q3 + 1.5 * iqr)]
print("beyond 1.5 IQR:", len(outliers))
print(outliers.head())

Duplicates are cheap to check and expensive to find later (a duplicated test row inflates the score). Outliers get inspected row by row - one extreme value is usually a data error to fix, a whole tail is a real population to keep. The decision is data-specific, and EDA is where it gets made, not inside the model.

The EDA Checklist

Row/column counts and the describe table - what is here.
Missing values per column - what is absent.
Distributions of the target and each important feature - the shapes.
Correlation with the target and between features - the signals and the redundancies.
Duplicates and extreme rows - the errors.
One written summary: the three things that matter most for modeling.

Summary

Run EDA as a fixed battery: describe, missing counts, distributions, correlations, group-by-target tables, duplicates, and the IQR tail - then write down what you found. The goal is not a report, it is a short list of decisions: what to impute, what to drop, what to encode, and which features the model should trust. Ten minutes of this work saves an afternoon of debugging a model that was built on a misunderstanding.