Design a Search & Ranking System
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Design a Search & Ranking System

Track

ML System Design

Inspired by

Airbnb / Amazon

Scale

100M+ listings · 10k+ QPS · <300ms budget

Difficulty

Intermediate

"Design search" looks like an information-retrieval question, but a modern search system is an ML ranking system with an IR front-end. Walk it through the ML project loop — the winning move is separating retrieval (cheap, high recall) from ranking (expensive, high precision), exactly like a feed. This domain is unusually well documented: Airbnb has published its ranking evolution end-to-end, Amazon its semantic retrieval system, and Facebook its embedding-based retrieval stack — we'll lean on all three.

1. Clarify the problem

  • What are we searching? A catalog of listings/products (structured fields + text + images), queried by a short, often ambiguous or misspelled query.
  • What is a good result? Relevance and the downstream goal — bookings / purchases, not just clicks. Search is judged on conversion, not lexical match.
  • Personalized? Yes — the same query means different things to different users (location, history, price sensitivity).
  • Scope: ranking + retrieval. Out of scope: the crawl/index infra, ads.

Interview signal

State early that relevance labels are implicit and biased (you only observe clicks/bookings on what you showed, in the order you showed it) — position bias is the defining data problem of search ranking.

Think firstIn a marketplace, the searcher isn't the only party — a host or seller can decline the booking. What does 'relevant' mean now, and what extra label does your ranker need?Commit to an answer, then reveal

Relevance becomes two-sided: the best result is one the guest wants and the host will accept. That means host rejections are labels too — Airbnb feeds host-rejection signals into its search embeddings so listings whose hosts decline this kind of guest get demoted. A guest-side-only ranker fills the top slots with bookings that will bounce off a rejection, which is worse than a weaker match that completes.

2. Requirements & scale

DimensionTargetConsequence
Latency< 300 ms p99Two-stage funnel; ANN + a bounded ranker
Catalog100M+ itemsInverted index + embedding index, not scans
Throughput10k+ QPSCheap retrieval, GPU/optimized ranker on ~1k candidates
RobustnessTypos, synonyms, multilingualQuery understanding + semantic retrieval
FreshnessNew/updated items searchable in minutesIncremental indexing
Think firstYou already have BM25 over an inverted index — a classic search engine. What can't it do, and where exactly does ML enter?Commit to an answer, then reveal

Lexical match can't bridge vocabulary ("beach house" vs "oceanfront villa"), can't personalize, and doesn't know that relevance ≠ conversion. ML enters in three slots: query understanding, a semantic retrieval channel alongside BM25, and a learned ranker on top — not as a replacement for the index but as stages around it.

3. ML problem framing

Search is a pipeline of ML slots:

  1. Query understanding: spell-correct, tokenize, detect intent/entities, expand synonyms. Framed as tagging + classification.
  2. Retrieval (hypothesis space): fetch ~1000 candidates via hybrid lexical (BM25 over an inverted index) + semantic (embedding ANN) recall.
  3. Ranking (hypothesis space + objective): a learning-to-rank model scores candidates on P(book | query, item, user).

What was breaking before? Pure lexical match (BM25) misses semantics ("beach house" vs "oceanfront villa") and can't personalize; a single ranker over the whole catalog blows the latency budget. The funnel fixes both. Amazon's product search team catalogued exactly where lexical match fails — hypernyms and synonyms, morphological variants ("woman" vs "women"), spelling errors — and reported a +4.7% Recall@100 lift from adding a trained semantic-matching channel (Nigam et al., KDD 2019). Facebook Search ran on Boolean lexical matching for years before adding embedding-based retrieval alongside (not instead of) the inverted index (Huang et al., KDD 2020).

Think firstYou're building the ranker's training set. Clicked items are the positives — what should the negatives be?Commit to an answer, then reveal

Not random catalog items: separating "beach house" from a lawnmower is too easy and teaches the model nothing. Use hard negatives — items shown in the same session that the user skipped, so the model learns to resolve close calls. And remember the positives are biased by position: the user clicked partly because you ranked it first.

4. Data & features (spine: data)

  • Labels: clicks, bookings/purchases, dwell — graded relevance. De-bias with inverse-propensity weighting or a click model to correct for position.
  • Features: query (embedding, length, intent), item (text/image embeddings, price, quality, popularity), query–item (BM25 score, semantic similarity, historical CTR), and user context.
  • Hard negatives: sample non-clicked items shown in the same session — the model must learn to separate close calls, not just random junk. Amazon's semantic-search loss makes this explicit with three tiers — purchased, impressed-but-not-purchased, and random, so the model learns both the coarse and the fine distinction.
  • Position bias: treat the probability that a user examined a position as a propensity and inverse-weight the logged clicks — the counterfactual learning-to-rank recipe (Joachims et al., WSDM 2017).
Deep dive
BM25 — the lexical retrieval baseline
Includes runnable notebook
Think firstAirbnb rode a GBDT ranker for years until its gains plateaued, then went neural. What is the first neural network you'd actually ship — and why is 'a deep net on all the features' the wrong answer?Commit to an answer, then reveal

Airbnb's published answer: a single-hidden-layer network on the same features as the GBDT, shipped mainly to prove the training-serving pipeline end to end, then a lambdarank NN, then deeper nets. Going straight to a deep architecture stacks two risks (new plumbing and new hypothesis space), so when the metric drops you can't tell which broke. Most of their paper is failed "obvious" ideas; the pipeline-first discipline is what let them find the ones that worked.

5. Model (spine: hypothesis space + objective + optimization)

Retrieval: two-tower + inverted index. A query tower and an item tower share an embedding space; item embeddings are precomputed and indexed in an ANN structure, merged with BM25 candidates. Personalization can ride on the same machinery: Airbnb learns listing embeddings from click sessions (skip-gram over co-clicked listings, with the booked listing as a global positive) and uses the user's recent-click embeddings as real-time ranking features — the search-and-similar-listings channels they personalize this way drive 99% of conversions (Grbovic & Cheng, KDD 2018).

Ranking: learning to rank. Start with a GBDT — LambdaMART is still a production workhorse and won the Yahoo! Learning to Rank Challenge (Burges, MSR-TR-2010-82) — on the query–item features; graduate to a pairwise/listwise neural ranker optimizing NDCG. Baseline first: BM25-only, to measure the lift retrieval and the ranker each add.

Deep dive
HNSW — the graph index behind semantic retrieval
Includes runnable notebook
Think firstBM25 scores and embedding cosine similarities live on incomparable scales, and the union of both channels is ~1000 candidates. Who decides the final order — and what role do the retrieval scores still play?Commit to an answer, then reveal

The ranker decides; retrieval scores select, they don't rank. Naively blending a BM25 score with a cosine similarity is arbitrary — the robust pattern is to pass both scores in as features of the learning-to-rank model, which is exactly what Facebook does with its ANN similarity in embedding-based retrieval. Retrieval's only job is to make sure the right answer is somewhere in the thousand.

6. Serving & scaling

StageBudgetHow
Query understanding~20 msCached, small models
Retrieval (BM25 ∪ ANN)~60 msInverted index + HNSW, merged & deduped
Ranking~120 msGBDT/neural ranker on ~1000 candidates
Blend + business rules~30 msDiversity, availability, policy

Cache popular-query results; precompute item embeddings; degrade to BM25-only if the ranker times out.

Deep dive
Vector databases — serving the ANN index
Includes runnable notebook
Think firstYou have two candidate rankers and limited live traffic. How do you find the better one without burning weeks of A/B tests?Commit to an answer, then reveal

Interleaving: blend both rankers' results into one list and see whose items get clicked — far more statistically sensitive than a traffic-split A/B, so it works as a cheap screen. Offline NDCG on de-biased labels filters first; interleaving picks the promising challenger; the full A/B on bookings makes the final call.

7. Evaluation (spine: evaluation)

  • Offline: NDCG@k, MRR, MAP on graded relevance; Recall@k for retrieval. Use interleaving to compare rankers sensitively before a full A/B.
  • Online: A/B on the business metric (bookings/purchases per search), with guardrails: zero-result rate, latency, query-abandonment.
  • Human relevance judgments for a gold set the implicit metrics can't cover.
Think firstBookings — your online metric — lag searches by days to weeks. What signals warn you within hours that search quality regressed, without waiting for any labels?Commit to an answer, then reveal

Behavioral proxies that need no labels: zero-result rate, query reformulation rate, abandonment, click-position distribution, and latency. A user who immediately re-types a longer query is telling you the first result page failed — that signal lands in your logs within minutes, weeks before the booking numbers move. Alert on these; treat the lagging business metric as the verdict, not the alarm.

8. Monitoring & feedback (spine: feedback)

  • Drift: query mix and inventory shift seasonally; monitor zero-result and reformulation rates as early warnings.
  • Retraining: rankers retrain frequently on fresh click logs; the loop closes back into data, with debiasing to avoid reinforcing prior exposure.

9. Tradeoffs & alternatives

ChoiceAlternativeWhy
Hybrid lexical + semanticPure semantic (embeddings only)Lexical still wins on exact IDs/SKUs and rare terms; hybrid is robust
Two-stage funnelOne ranker over all itemsLatency makes full scoring impossible
GBDT rankerDeep listwise ranker firstGBDT is a strong, cheap, interpretable baseline; add depth if it pays
IPS debiasingIgnore position biasIgnoring it makes the model relearn its own exposure

10. Interviewer follow-ups

A user searches an exact product SKU and gets fuzzy semantic matches. Fix?

Hybrid retrieval with a lexical exact-match channel that can short-circuit: if a high-confidence exact ID/SKU match exists, boost or pin it. Pure embedding retrieval blurs exact identifiers — that's why lexical stays in the funnel.

How do you handle a brand-new listing with no click history?

Content-based features (text/image embeddings, structured attributes) let it be retrieved and ranked on day one; an exploration boost gives it initial exposure to gather signal. This is the search flavor of the cold-start problem.

Offline NDCG improves but bookings drop in the A/B. Why?

The offline metric rewards clicks that don't convert (clickbait titles/images), position bias inflates it, or there's train-serve feature skew. Optimize toward the booking label, not clicks, and interleave before trusting offline NDCG.

Key takeaways

  • Modern search is retrieval + learning-to-rank, not string matching.
  • Hybrid lexical + semantic retrieval beats either alone; lexical keeps exact matches, semantic adds meaning.
  • Optimize the downstream conversion, correct position bias in the labels, and let interleaving/A-B (not offline NDCG) decide.

References