The Recommendation ProblemOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

The Recommendation Problem

Every feed, store, and streaming service answers the same question millions of times a second: of all the items, which few should I show this user now? Recommender systems are the ML behind that choice, and they're among the highest-value ML applications in industry. This lesson frames the problem, contrasts the two classic families of solution, and defines how we actually measure a recommendation. Recommenders stress the far end of the ML project loop: getting the objective and ranking evaluation right (recall@k, NDCG) matters as much as the model, and because recommendations shape the very clicks you train on next, the deployment-feedback loop is unusually tight.

Deep dive
The ML Project Loop
Includes runnable notebook

Real-world examples: Netflix's "Top picks," Amazon's "Customers also bought," Spotify's Discover Weekly, YouTube's up-next, and the ranking of any social feed.

The setup

You have a set of users UU and items II, and a sparse record of past interactions. The goal is to predict, for a user, the items they'd most likely engage with, and surface a short ranked list.

The interaction data is a user–item matrix R\mathbf{R}, where RuiR_{ui} is user uu's response to item ii. Crucially, this matrix is extremely sparse: a user has touched a tiny fraction of all items, so most entries are unknown (not zero). Recommendation is, at heart, filling in those unknowns.

Explicit vs. implicit feedback

ExplicitImplicit
Signalratings, thumbs, reviewsclicks, watches, purchases, dwell time
Meaningstated preferenceinferred preference
Volumesparse, costlyabundant, cheap
Negativesa low rating is real dislikea non-click is ambiguous (unseen? disliked?)

Implicit feedback has no true negatives

A user not clicking an item usually means they never saw it — not that they dislike it. So you can't treat un-interacted items as confirmed negatives. Implicit-feedback models use techniques like negative sampling and confidence weighting (interactions are positive signals of varying strength) rather than assuming missing = negative.

Two classic families

Content-based filtering recommends items similar to ones the user liked, using item features (genre, text, tags). Build a profile vector from a user's history, then rank items by feature similarity (e.g. cosine).

  • ✅ Works for brand-new items (uses their features) and needs no other users.
  • ❌ Stuck in a "filter bubble" (only ever recommends more of the same); needs good item features.

Collaborative filtering (CF) recommends what similar users liked, using only the interaction matrix — no item features at all. "Users who liked what you liked also liked X."

  • ✅ Discovers surprising cross-domain recommendations; no feature engineering.
  • ❌ Suffers cold start: a brand-new user or item has no interactions to learn from.

Memory-based vs. model-based CF

Early CF was memory-based: find the k nearest users (or items) by interaction overlap and average their preferences — essentially KNN on the rating matrix. Modern CF is model-based: learn low-dimensional latent factors that generate the matrix (matrix factorization, next lesson). Production systems usually hybridize content + collaborative signals to get the best of both and soften cold start.

The cold-start problem

Collaborative filtering can't recommend for entities with no interaction history:

  • New user: no history to find similar users. Mitigate with onboarding questions, popularity defaults, or content/context features.
  • New item: no one has interacted with it. Mitigate with content-based features until interactions accumulate.

Cold start is the reason pure CF is rarely deployed alone.

Measuring a recommender: ranking metrics

A recommender outputs a ranked list, so accuracy-style metrics miss the point — what matters is whether relevant items appear near the top. The standard top-kk metrics:

  • Precision@k: fraction of the top-kk recommendations that are relevant.
  • Recall@k: fraction of all relevant items that appear in the top-kk.
  • NDCG@k (Normalized Discounted Cumulative Gain) — rewards putting relevant items higher, with a logarithmic position discount:
DCG@k=∑i=1krelilog⁡2(i+1),NDCG@k=DCG@kIDCG@k\text{DCG@}k = \sum_{i=1}^{k} \frac{\text{rel}_i}{\log_2(i+1)}, \qquad \text{NDCG@}k = \frac{\text{DCG@}k}{\text{IDCG@}k}

where IDCG\text{IDCG} is the DCG of the ideal (perfectly sorted) ranking, so NDCG ∈[0,1]\in [0,1]. MRR (mean reciprocal rank) and MAP are also common.

Worked example. Top-3 list where items 1 and 3 are relevant (rel=1\text{rel} = 1) and item 2 is not: DCG@3=1log⁡22+0+1log⁡24=1+0.5=1.5\text{DCG@3} = \tfrac{1}{\log_2 2} + 0 + \tfrac{1}{\log_2 4} = 1 + 0.5 = 1.5. The ideal ordering puts both relevant items first: IDCG@3=1+1log⁡23=1.631\text{IDCG@3} = 1 + \tfrac{1}{\log_2 3} = 1.631. So NDCG@3=1.5/1.631=0.92\text{NDCG@3} = 1.5 / 1.631 = 0.92 — the misplaced second slot costs 8%.

Why position-aware metrics?

Two recommenders can have identical recall@10 yet very different user experience if one puts the relevant item at rank 1 and the other at rank 10. NDCG and MRR capture that the order within the list matters — the top slot is worth far more than the tenth.

Five metrics, one list

P@31.000
P@100.300
MRR1.0000
MAP1.0000
NDCG@101.0000

Press the first two presets in turn. P@10 does not move — it counts a set, and the set is identical — while NDCG falls from 1.0000 to 0.3452 and MRR from 1.0000 to 0.1250. Same documents, same recall, a ranking nobody would ship.

Click a row to cycle its relevance grade (0–3). Every metric is recomputed from the grades: DCG = Σ (2^rel − 1)/log₂(i+2) with NDCG normalising by the ideal ordering, MAP averaging precision at each hit, MRR reading only the first. Bar length is the item's gain; position is its rank.

That callout is a claim you can make numeric in two clicks. Press relevant at the top, then same items, at the bottom — the list contains exactly the same three documents, so precision and recall cannot tell them apart:

P@10P@3MRRMAPNDCG@10
relevant at the top0.3001.0001.00001.00001.0000
same items, at the bottom0.3000.0000.12500.21570.3452

P@10P@10 does not move at all. NDCG falls 2.90×2.90\times and MRR 8×8\times. A metric that counts a set is blind to the only thing a ranker controls, which is why "we improved recall@100" is not by itself evidence that anyone's feed got better.

Reading the gains column

Each row's contribution is (2rel−1)/log⁡2(i+2)(2^{\text{rel}} - 1)/\log_2(i+2): the numerator makes a grade-3 item worth 7 and a grade-1 item worth 1, and the denominator discounts by position. Moving a perfect item from rank 1 to rank 8 keeps its relevance and divides its contribution by 3.17.

Exercise

You train a recommender on click logs (implicit feedback). Why is it wrong to treat every item a user did NOT click as a negative example?

Need a hint?

Did the user even see those items?

Exercise

Pure collaborative filtering cannot make good recommendations for a brand-new item with no interactions. What is this problem called, and a standard fix?

Need a hint?

It also affects brand-new users.

Exercise

Two recommenders both place exactly the same 3 relevant items somewhere in their top-10, so they have identical recall@10. Why might their NDCG@10 differ?

Need a hint?

What does the log discount in DCG reward?

Common mistakes

  1. Treating missing implicit feedback as a negative. A non-click is ambiguous; use negative sampling, not "missing = dislike."
  2. Evaluating with RMSE only. Rating-prediction error ignores ranking quality; report top-kk ranking metrics (recall@k, NDCG).
  3. Ignoring cold start. A pure-CF demo looks great offline but fails the moment a new user or item arrives.
  4. Leaking the future. Splitting interactions randomly lets the model "see" later behavior; split by time to mimic deployment.

Key takeaways

  • Recommendation = filling in a sparse user–item matrix and surfacing a short ranked list.
  • Explicit feedback (ratings) is sparse but clear; implicit feedback (clicks) is abundant but has no true negatives.
  • Content-based filtering uses item features (handles new items, risks filter bubbles); collaborative filtering uses the interaction matrix (discovers novel items, suffers cold start). Production systems hybridize.
  • Judge recommenders with position-aware ranking metrics (recall@k, NDCG@k, MRR), not plain accuracy or RMSE.

Related concepts

  • Matrix Factorization: the model-based CF workhorse that learns latent user/item factors
  • KNN: the nearest-neighbor idea behind memory-based collaborative filtering
  • Classification Metrics: precision/recall, which the top-k ranking metrics extend
  • The Cold Start Problem: strategies for new users, items, and contexts with no interaction history

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
The Recommendation ProblemLinear Algebra for MLMatrix FactorizationK-Nearest NeighborsClassification & Regressi…The ML Project LoopThe Cold Start Problem in…