Recommender Systems

Recommender systems predict which items a user will value: products on an e-commerce store, articles on a news feed, courses on a learning platform. They drive measurable engagement because they replace a blind catalog with a personalized shortlist. Two families dominate. Content-based filtering recommends items similar to what the user already liked, using item attributes. Collaborative filtering ("people who liked X also liked Y") learns from the behavior pattern itself, and it usually performs better once enough interaction data accumulates.

User-Item Interaction Matrix

Collaborative filtering starts from an interaction matrix: rows are users, columns are items, cells hold ratings or implicit signals such as clicks and purchases. In pandas, a pivot builds it; most cells are empty, which is exactly the signal the model must fill in.

import pandas as pd

inter = pd.read_parquet("interactions.parquet")
R = inter.pivot_table(index="user_id", columns="course_id",
                      values="rating", aggfunc="mean")
print(R.shape, "dense; density = %.4f" % (R.notna().mean().mean()))

Item-Based Similarity

The simplest collaborative model: two items are similar if the same users interacted with both. Cosine similarity over item columns yields a nearest-neighbor index that generalizes "related items" cheaply and explains its choices naturally.

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

Rf = R.fillna(0)
sim = cosine_similarity(Rf.T)                     # item x item
sim_df = pd.DataFrame(sim, index=R.columns, columns=R.columns)

def similar(course_id, n=5):
    return sim_df[course_id].drop(course_id).nlargest(n)

print(similar(1721))

Matrix Factorization

Matrix factorization decomposes R into two low-rank matrices, U (user factors) and P (item factors), so the dot product of a user row and an item row is the predicted rating. The product U @ P reconstructs the full matrix including the empty cells — that is the recommendation. Truncated SVD works on mean-centered ratings; alternating least squares handles implicit feedback.

from sklearn.decomposition import TruncatedSVD
from sklearn.impute import SimpleImputer
import numpy as np

X = SimpleImputer(strategy="constant", fill_value=0).fit_transform(Rf)
Xc = X - X.mean(axis=0)  # center; missing entries become "average"

svd = TruncatedSVD(n_components=20, random_state=42)
U = svd.fit_transform(Xc)
pred_all = U @ svd.components_ + X.mean(axis=0)

uid = 0
scores = pd.Series(pred_all[uid], index=R.columns)
already = Rf.columns[Rf.iloc[0] > 0]
print(scores.drop(already).nlargest(3))  # top-3 unseen picks

Content-Based Filtering

Content-based systems need item metadata instead of behavior. Vectorize descriptions with TF-IDF, then recommend items whose vectors sit closest to the vector of the item the user is looking at. It works from day one for brand-new items, which collaborative methods cannot match.

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(stop_words="english", max_features=5000)
V = tfidf.fit_transform(courses["description"])
target = courses.index[courses["course_id"] == 3009][0]
sims = (V @ V[target].T).toarray().ravel()
best = np.argsort(-sims)[1:6]
print(courses.iloc[best][["course_id", "title"]])

Evaluation and Hybrid Systems

Split interactions chronologically: train on older behavior, test on recent ones you deliberately hid, then rank test items per user and score with Precision@k, Recall@k, or NDCG@k. Coverage — what fraction of the catalog ever gets recommended — guards against filter bubbles. Production systems are hybrids: blend collaborative scores with content similarity, apply business filters, and re-rank with a small learned model.

Summary

Recommender systems turn an interaction matrix into personalized rankings. Item similarity gives fast explainable neighbors, matrix factorization with SVD reconstructs the whole matrix from latent factors, and TF-IDF content matching covers new items cold-start-free. Evaluate with time-aware splits and precision/NDCG@k, and ship one coherent hybrid rather than over-engineering a single golden algorithm.