NumPy and Pandas

A pandas Series and DataFrame are ndarrays wearing labels: the values live in NumPy blocks, and pandas adds an index (row labels), column names, and missing-value semantics on top. The interop is therefore seamless in one direction - the moment you drop the labels you are doing NumPy - and the discipline is knowing exactly when a pandas operation hands you back an ndarray and you have silently lost the index. This chapter is the boundary: how to move data between the two, and which idiom keeps you on the fast, labeled path.

What a DataFrame Actually Is

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "a": [1.0, 2.0, 3.0],
    "b": [4.0, 5.0, 6.0],
})
print(df.values)                 # the underlying ndarray (2, 2)
print(df["a"].values)            # one column as a 1-D ndarray
print(type(df.values))           # numpy.ndarray

# each column is an ndarray-backed Series
print(df["a"].to_numpy())        # the modern, explicit form

The column values are stored in NumPy arrays - one per dtype block, so a mixed int/float frame is several blocks. .to_numpy() (and the older .values) exposes that buffer. The key asymmetry: a pandas operation that preserves the index returns a Series/DataFrame; an operation that is fundamentally elementwise-on-values often returns a bare ndarray, and with it the index is gone. That is the first thing to check whenever a result stops behaving like a DataFrame.

Moving Data Between Them

import numpy as np
import pandas as pd

# ndarray in: columns become labeled Series
arr = np.arange(12, dtype=float).reshape(3, 4)
df = pd.DataFrame(arr, columns=["w", "x", "y", "z"])
print(df.shape, df.columns.tolist())

# labeled in: the index is preserved
df2 = pd.DataFrame({"a": [1, 2], "b": [3, 4]}, index=["r1", "r2"])
print(df2.loc["r2"])             # labeled row access

# ndarray out, labels dropped
v = (df["w"] + df["x"]).to_numpy()
print(type(v))                    # numpy.ndarray - no index now

# back in, with the index re-attached
recovered = pd.Series(v, index=df.index, name="w_plus_x")
print(recovered.index.tolist())   # the original row labels restored

The conversion is cheap - to_numpy is a view or a near-view, not a deep copy - so the pattern is: do the heavy elementwise work in NumPy (where the ufuncs and broadcasting are native), then re-wrap the result in a Series with the original index if you need the labels. The index is the thing that does not survive the trip; everything else (values, dtype, shape) does.

Where to Do the Computation

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "price": np.random.default_rng(0).normal(100, 10, 1000),
    "qty": np.random.default_rng(1).integers(1, 10, 1000),
})

# FAST: vectorized - pandas calls the NumPy engine on the blocks
df["revenue"] = df["price"] * df["qty"]
df["flag"] = np.where(df["price"] > 105, "high", "normal")

# SLOW: the Python loop, one row at a time - avoid it
# df["revenue"] = [p * q for p, q in zip(df["price"], df["qty"])]
# the list comprehension is correct but 10-100x slower at scale

The rule mirrors the pure-NumPy one: vectorize. pandas' elementwise operators (*, +, comparisons) dispatch to the NumPy engine on the underlying blocks, so df["a"] * df["b"] is as fast as the ndarray form - keep the labels, keep the speed. The np.where idiom slots directly into a column. The thing to avoid is the per-row Python loop (iterrows, list comprehensions over columns) - it is the same 50-100x gap as in pure NumPy, just with an extra index lookup per row.

The Chained-Assignment Trap

import numpy as np
import pandas as pd

df = pd.DataFrame({"a": [1.0, 2.0], "b": [3.0, 4.0]})

# chained indexing may return a COPY - the write is lost silently
# df["a"][0] = 99.0        # SettingWithCopyWarning - did it work?

# the safe forms: .loc (label) or .iloc (position), one step
df.loc[0, "a"] = 99.0
df.iloc[1, 1] = 44.0
print(df)                     # both writes applied, no ambiguity

Chained indexing - df["a"][0] - selects a column, then indexes into the result, and the intermediate step may return a copy rather than a view. Writing through a copy changes the temporary and leaves the original frame untouched, with only a warning (or, in newer pandas, an error) to show it. The fix is a single-step selection: df.loc[label, column] for label-based or df.iloc[row, col] for position-based, each of which writes straight into the frame. When a write "did not take effect," this is the first thing to check.

When to Use Which

The clean mental model: pandas is the labeled, missing-value-tolerant table layer; NumPy is the computation layer underneath. Use pandas to load, label, clean, and group; drop to NumPy for the dense numerical core; and re-wrap with the index whenever the result needs to stay in table form. The two are not competing - one is built on the other - and the skill is moving between them without losing the index or accidentally looping row by row.

Summary

Treat a DataFrame as ndarrays plus an index: the values are NumPy blocks, exposed by to_numpy(), and most elementwise operators already dispatch to the NumPy engine, so df["a"] * df["b"] is as fast as the array form. Do the dense math vectorized (NumPy idiom or pandas operators), keep per-row Python loops out of the hot path, write with single-step loc/iloc to dodge the chained-assignment copy, and remember that the index - not the values - is what gets dropped when a result comes back as a bare ndarray. Re-wrap with the original index when the table form must survive.