Deep & Two-Tower Recommenders
Matrix factorization is linear and tied to a fixed set of users and items. Modern industrial recommenders are neural: they learn rich embeddings from features, model nonlinear interactions, and (crucially) are built to retrieve from billions of items in milliseconds. This lesson covers the architecture that makes web-scale recommendation work: the two-tower model, and the retrieval → ranking funnel it sits in.
Real-world examples: YouTube's candidate generation, Google Play and the TensorFlow Recommenders two-tower models, Pinterest's PinSage, and essentially every large-scale "for you" feed.
Neural collaborative filtering
The first step beyond matrix factorization: keep the learned user/item embeddings, but replace the fixed dot product with a learned function. Concatenate the embeddings (plus any side features) and feed them through an MLP:
This adds nonlinearity (the MLP can model interactions a dot product can't) and lets you fold in side features (user demographics, item metadata, context like time of day) — directly softening cold start, since a brand-new item still has features even with no interactions.
Expressive, but you can't score billions of items with it
An MLP over the concatenated pair must be evaluated per candidate item. Scoring every one of a billion items for every request is impossible in real time. This is the central scaling problem, and why the two-tower architecture exists.
The two-tower architecture
A two-tower (dual-encoder) model keeps the user and item encoders separate so the expensive item work can be done offline:
- A user/query tower maps the user + context to an embedding .
- An item/candidate tower maps each item to an embedding in the same space.
- The score is a simple dot product (or cosine) .
Because the towers don't interact until the final dot product, you can precompute every item embedding offline and store them in a vector index. At request time you embed only the user once, then find the nearest item vectors.
Two-tower = matrix factorization, generalized
Replace each tower with a plain embedding lookup and the two-tower model is matrix factorization. The towers generalize it: they're arbitrary networks over features, so they handle cold start and context, while keeping MF's all-important property that scoring is a dot product in a shared space.
Training: in-batch negatives
Two-tower retrieval models are trained with a contrastive objective: pull a user toward the items they engaged with, push them away from items they didn't. The efficient trick is in-batch negatives — for a batch of positive (user, item) pairs, treat the other items in the batch as negatives for each user, via a softmax over in-batch dot products:
This is the same contrastive idea as CLIP/embedding training — one forward pass yields positives and negatives for free.
Retrieval then ranking
No single model both scans billions of items and scores each one richly. Production recommenders are a funnel:
| Stage | Model | Candidates | Goal |
|---|---|---|---|
| Retrieval | two-tower + ANN | billions → ~hundreds | cheap, high-recall shortlist |
| Ranking | heavy cross-feature model (e.g. NCF / GBDT / DLRM) | hundreds → ~10 | precise ordering of the shortlist |
| (Re-ranking) | business rules, diversity | ~10 | freshness, dedup, fairness |
Retrieval optimizes recall (don't miss good items) cheaply; ranking optimizes precision/NDCG (get the order right) expensively, but only on the small shortlist where it's affordable.
Which stage is actually holding you back
Which stage of the funnel is actually holding you back
Cheap over everything, expensive over a few
That is the whole reason two-stage systems exist: you cannot afford the good model on the whole catalogue. But it introduces a constraint that is easy to forget.
Move K and watch the second stage's workload — and its ceiling — move together.
the funnel
3,000 → 100 → 10
ranker does 3.3% of the scoring work
3,000 items per user. Retrieval scores all of them cheaply and keeps its top K; the ranker scores only those K accurately and returns the top 10. Both see the same ground truth through their own independent noise. Every recall figure below is simulated and measured here.
The funnel introduces one constraint that is easy to forget: the ranker can only reorder what retrieval handed it. Anything the first stage dropped is gone, whatever the second stage's quality — set the ranker to perfect in the walkthrough and end-to-end recall lands exactly on the retrieval ceiling and stops.
That splits the system into two regimes. At small , end-to-end sits right on the ceiling: retrieval is the binding constraint and the ranker is idle capacity. Past roughly in this simulation the ceiling reaches 1.000 while end-to-end plateaus near 0.81 — retrieval has stopped losing anything and the ranker's own error is now the entire limit.
Which regime you are in decides what an investment is worth. Improving the ranker from noise 0.20 to perfect buys +0.061 at and then stops dead against the ceiling; the identical upgrade at buys +0.227, nearly four times as much. Holding a perfect ranker at and halving retrieval's noise instead takes end-to-end from 0.665 to 0.989.
This is the mechanical explanation for a very common experience: the ranker improved in offline eval and the end-to-end metric did not move, because the system was never ranker-bound, and nothing in the ranker's own metrics would have told you that. Measure the ceiling separately from the final number, and you can see which stage owns your loss.
Serving with approximate nearest neighbors
The retrieval step ("find the item vectors closest to this user vector") is exactly the ANN search problem. Item embeddings live in a vector index (HNSW/IVF); a query embedding is matched against them in milliseconds.
This is why two-tower retrieval and vector databases are inseparable: the architecture is designed so retrieval reduces to nearest-neighbor lookup over precomputed item vectors.
Exercise
Why does a two-tower recommender keep the user and item encoders separate, scoring only with a dot product at the end?
Need a hint?
What can you do offline if the towers never interact until the final dot product?
Exercise
Industrial recommenders use a retrieval stage then a ranking stage. What is each stage optimized for?
Need a hint?
Billions → hundreds, then hundreds → ten.
Exercise
Neural collaborative filtering feeds concatenated user/item embeddings (plus side features) through an MLP. What does this add over plain matrix factorization, and what does it cost?
Need a hint?
More expressive — but can you precompute item scores?
Common mistakes
- Using a cross-feature MLP for retrieval. It can't precompute item embeddings, so it can't scan billions of items — use it for ranking the shortlist, not retrieval.
- Forgetting the two towers must share an embedding space. User and item vectors are only comparable by dot product if trained jointly into the same space.
- Sampling negatives poorly. All-random negatives are too easy; in-batch (and hard) negatives give a far stronger training signal.
- Skipping the ranking stage. Retrieval alone gives a high-recall but loosely-ordered list; a ranker is what makes the top few great.
Key takeaways
- Neural CF replaces MF's dot product with an MLP over concatenated embeddings + side features — more expressive and cold-start-friendly, but must be scored per item.
- The two-tower model keeps user and item encoders separate so item embeddings are precomputed offline and scoring stays a dot product in a shared space — MF, generalized to features.
- Train retrieval with a contrastive in-batch-negative objective; it's the CLIP-style trick reused for recommendation.
- Web-scale recsys is a retrieval (two-tower + ANN, high recall) → ranking (heavy model, high precision) funnel; retrieval is nearest-neighbor search over item vectors.
Related concepts
- Matrix Factorization: the linear dot-product model the two towers generalize
- Embeddings & Semantic Search: the embedding + ANN retrieval the item tower relies on
- Training Embedding Models: the contrastive in-batch-negative objective, shared with two-tower training