Loading Data

Every model is only as good as the data that reaches it, and the data has to get in first. pandas is the standard loader for tabular data - CSV, Excel, Parquet, JSON, and database tables all come in as a DataFrame, the table-like structure the rest of the tutorial operates on. This chapter covers the common sources and the loading options that prevent the classic first-day problems: wrong types, mystery columns, and memory blowups.

CSV Files

import pandas as pd

df = pd.read_csv("sales.csv")
print(df.shape)        # (rows, columns)
print(df.head())       # first five rows
print(df.dtypes)       # the inferred type of every column

Always run head and dtypes immediately after loading - half of all "weird model behavior" starts with a column that pandas guessed wrong (a number read as text, a date read as an integer). The options that matter most:

df = pd.read_csv(
    "sales.csv",
    sep=";",                     # not every CSV uses commas
    na_values=["N/A", "none", ""],   # treat these as missing
    parse_dates=["created_at"],      # parse to datetime at load time
    usecols=["order_id", "amount", "created_at"],   # skip the rest
    dtype={"customer_id": str},     # force a type (e.g. leading zeros)
)

Other Common Formats

df_excel = pd.read_excel("report.xlsx", sheet_name="Q3")
df_parquet = pd.read_parquet("events.parquet")     # fast, typed, compressed
df_json = pd.read_json("records.json")
df_sql = pd.read_sql("SELECT * FROM orders", con=engine)   # via SQLAlchemy

Parquet is the best default for anything larger than a few hundred MB: columnar, typed, compressed, and far faster than CSV for repeated loading. Excel is for hand-made reports (and its sheet names matter). SQL reads pull straight from a database, which is how production pipelines usually start.

Small and Synthetic Data

from sklearn.datasets import fetch_california_housing, load_iris

iris = load_iris(as_frame=True)     # a classic 150-row labeled dataset
X = iris.frame[["sepal length (cm)", "sepal width (cm)",
                "petal length (cm)", "petal width (cm)"]]
y = iris.frame["target"]

housing = fetch_california_housing(as_frame=True)   # a larger real dataset

The scikit-learn dataset loaders are the right tool for examples and unit tests: they need no files, no licenses to track, and no network in CI (they cache locally). When working on a real project, the "data" is your CSV or your database, and the loaders above are the front door.

Keeping an Eye on Memory

print(df.memory_usage(deep=True).sum() / 1e6, "MB")

# cut it: downcast numeric types and categorize strings
for col in df.select_dtypes("int64").columns:
    df[col] = pd.to_numeric(df[col], downcast="integer")
for col in df.select_dtypes("object").columns:
    if df[col].nunique() / len(df) < 0.5:
        df[col] = df[col].astype("category")

Large DataFrames usually want category dtype for low-cardinality string columns and downcasted integers - often a 3-5x memory reduction with no information loss. If the dataset does not fit in memory at all, the tool changes (dask, Polars, or SQL) and pandas is no longer the right loader.

Summary

Load with the format-appropriate pandas reader, inspect head and dtypes before anything else, and use the targeted options - na_values, parse_dates, usecols, dtype - to fix problems at the door. Prefer Parquet for large or repeated loads, scikit-learn loaders for examples and tests, and check memory_usage before the dataset decides to live on the swap partition.