Design a Visual Search System
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Design a Visual Search System

Track

ML System Design

Inspired by

Pinterest Lens / Amazon

Scale

billions of images · <300ms · embedding ANN

Difficulty

Intermediate

Visual search ("point your camera at a couch, find ones to buy") is a representation-learning + nearest-neighbor retrieval problem. There's no text query — the image is the query — so the whole system hinges on learning an embedding where visually/semantically similar items are close, then searching it fast. Pinterest, eBay, and Alibaba have all published their production systems — Pinterest's 2015 paper is famous for showing a small team with open-source tools and commodity cloud infra could build one, and that visual search measurably lifted engagement in live A/Bs (Jing et al., KDD 2015). Walk it through the ML project loop.

1. Clarify the problem

  • Goal: given a query image (often a messy real-world photo), return visually similar catalog items, ideally shoppable.
  • Similarity of what? Usually fine-grained product similarity (same style of shoe), not just category — the label definition drives everything.
  • Detect-then-search: a photo may contain several objects; often detect the object of interest first, then search on the crop.
  • Scope: embedding + retrieval + re-rank. Adjacent: the detection model, which we reuse.

Interview signal

The core question is "what does similar mean, and how do I get labels for it?" — exact-match (same product) and visually-similar are different objectives with different mining strategies. Pin this before choosing a loss.

Think firstOne surface needs exact-product matches ('buy this lamp'), another needs style-similar browsing, a third needs shop-the-look. Three embeddings or one — and what changes at the data level?Commit to an answer, then reveal

Pinterest tried both and consolidated: separate specialized embeddings became an operational tax (three models to train, serve, and keep consistent), so they moved to one unified embedding trained multi-task across all three objectives, which matched or beat the specialists because the tasks share signal. The data-level consequence: each task contributes its own labeled pairs (engagement for browse, human-verified exact matches for shop), and the sampling mix across tasks becomes a first-class tuning knob.

2. Requirements & scale

DimensionTargetConsequence
Latency< 300 msPrecomputed embeddings + ANN, not pairwise compare
CatalogBillions of imagesHNSW / IVF-PQ index, quantized embeddings
RobustnessReal-world photos (lighting, angle, clutter)Heavy augmentation; detect-then-crop
FreshnessNew products searchable in minutesIncremental embedding + index updates
Think firstThere's no text query — the photo is the query. What does the whole system reduce to, and what single artifact caps its quality?Commit to an answer, then reveal

It reduces to learning a distance: map every image to a vector such that "similar" means "nearby," then nearest-neighbor search does the rest. That makes the encoder the whole game — retrieval quality is bounded by embedding quality, and no amount of index engineering can recover similarity the embedding failed to encode.

3. ML problem framing

Metric / representation learning: train an encoder so that similar images map to nearby vectors, then do approximate nearest neighbor retrieval + a re-ranking pass. Retrieval quality is bounded by embedding quality — the model is the embedding.

What was breaking before? Classification tags are too coarse (they can't express "this specific mid-century chair"); pairwise image comparison at query time is computationally impossible at billions of items.

4. Data & features (spine: data)

  • Labels / pairs: positive pairs from the same product (different photos, angles), user co-engagement (clicked-together items), and human similarity judgments. Negatives from hard-negative mining (visually close but wrong).
  • Self-supervision: augmentations (crop, color, rotate) generate positive pairs cheaply — powerful when labels are scarce.
  • Multimodal: vision-language encoders let text and image share a space (search by photo, refine by words).
Think firstFor contrastive training, which examples matter more — the positives or the negatives?Commit to an answer, then reveal

The negatives — specifically the hard ones. Random negatives (a shoe vs. a sofa) are separated after one epoch and teach nothing; the model's final quality comes from learning to push apart items that are visually close but wrong (the same chair in a different style). Hard-negative mining is where the metric actually gets learned.

Deep dive
Vector databases — storing & serving embeddings
Includes runnable notebook

5. Model (spine: hypothesis space + objective)

  • Encoder: a CNN/ViT backbone fine-tuned with a contrastive / triplet loss (metric learning), or a self-supervised (SimCLR/DINO-style) then fine-tuned encoder.
  • Objective: pull positives together, push hard negatives apart — the choice of which negatives dominates final quality. Train multi-task across surfaces: Pinterest's unified embedding jointly optimizes browse, search, and shop objectives in one network, and binarizes the output embedding for cheap storage and retrieval without measurable precision loss (Zhai et al., KDD 2019).
  • Multimodal option: contrastive image–text pretraining at scale (CLIP trained on 400M web pairs) yields a joint space where photos and words are directly comparable (Radford et al., ICML 2021) — the foundation for "this, but in blue."
  • Re-ranker: a heavier model (or cross-attention over query+candidate) refines the ANN top-k on price, availability, and finer similarity.

CLIP: a shared image+text embedding space

‖z‖ = 1img1img2img3img4img5img6txt1txt2txt3txt4txt5txt6image embeddingtext embeddingzero-shot similarity (from selected text)
symmetric CLIP loss0.618
image→text top-1 acc100%
τ0.30
zero-shot pick for txt1img1 (cat)

Filled circles are images (one each for cat, dog, car, tree, beach, building); open squares are the paired text captions ("a photo of a …"). Both encoders project into the same plane; embeddings are L2-normalised so they live on the unit circle. Click any text square to see the zero-shot classification it would produce — dashed lines fan out to each image with opacity proportional to cosine similarity, and the top-1 image is highlighted. Drop τ and press play to watch symmetric contrastive descent pull each matched pair together.

A shared image–text embedding space (CLIP-style) is what lets a photo query and a catalog image land near each other, and enables "find this, but in blue."

Think firstA billion 512-dimensional float32 embeddings — roughly how much memory is that, and what does the number force?Commit to an answer, then reveal

512 × 4 bytes × 10⁹ ≈ 2 TB — too much to hold in RAM on a serving node, before you've even built an index. That's why billion-scale ANN pairs a graph or inverted-list index with product quantization, compressing each vector ~30–60× at a small recall cost, with an exact re-rank on the shortlist.

6. Serving & scaling

Offline: embed the whole catalog, quantize (PQ), build an ANN index — the IVF+PQ compressed-domain search popularized by Faiss is the standard recipe at billions of vectors (Johnson et al., 2017). Online: detect the object → crop → embed the query once → ANN search → re-rank → availability/business filters, all under 300ms. The detect-then-search step is production practice, not a nicety: Alibaba's Pailitao runs category prediction plus object localization before embedding, precisely because user photos are cluttered scenes, not catalog shots (Zhang et al., KDD 2018). Update the index incrementally as inventory changes.

Deep dive
HNSW — the graph index for billion-scale ANN
Includes runnable notebook
Think firstYou need an offline eval set for 'similar'. Clicks are abundant — why can't you just use click-through on results as ground truth, and what do you use instead?Commit to an answer, then reveal

Clicks conflate similarity with salience, position, and price — users click the eye-catching or cheap result, not the most similar one, and you only observe clicks on what the current embedding retrieved (selection bias). The honest setup layers three sources: human side-by-side judgments on a gold set for fine-grained similarity, known exact-match pairs (same product, different photos) for objective recall, and engagement as a directional online signal — never the offline ground truth.

7. Evaluation (spine: evaluation)

  • Offline: Recall@k / precision@k against similarity ground truth; mAP for retrieval. Embedding quality probes (does the space cluster products sensibly?).
  • Online: click-through and purchase rate on visual-search results, plus zero/low-result rate.
  • Human eval: side-by-side similarity judgments the implicit metrics miss.
Think firsteBay's catalog isn't a catalog — it's live listings that end, sell out, and appear by the million daily. What breaks in your beautifully precomputed index, and what's the redesign?Commit to an answer, then reveal

Staleness in both directions: results pointing at ended listings (visible breakage users punish immediately) and new inventory that's invisible until the next index build. A nightly rebuild can't keep up, so the eBay-style answer is a streaming ingest path — embed new listings on arrival, append to the index, tombstone dead ones and filter them at query time — with periodic full rebuilds to compact and retrain quantizers. Index freshness becomes a monitored SLA, just like latency.

8. Monitoring & feedback (spine: feedback)

  • Distribution shift: new product styles, seasonal inventory, and camera trends shift the query and catalog distributions; re-embed and retrain.
  • Feedback loop: engagement on returned items becomes new positive pairs, improving the embedding over time.

9. Tradeoffs & alternatives

ChoiceAlternativeWhy
Metric-learned embedding + ANNClassification + tag matchTags are too coarse for fine-grained visual similarity
Detect-then-crop-then-searchSearch the whole photoCluttered photos need the object isolated first
Quantized ANN (PQ)Exact NNExact search is infeasible at billions of vectors
Contrastive fine-tuneFrozen pretrained encoderDomain fine-tuning sharply improves in-catalog similarity

10. Interviewer follow-ups

The query photo has a couch, a rug, and a lamp. What do you return?

Detect objects first, let the user pick (or default to the most salient), then search on that crop. Embedding the whole cluttered scene averages the objects together and retrieves nothing well — detect-then-search is the fix.

How do you support 'find this dress but in red'?

A shared vision-language embedding space: embed the image, then compose with a text delta ("red") in the same space, or condition the query on an attribute. This is exactly what CLIP-style joint training buys you.

Recall@k is high offline but users don't click. Why?

The offline similarity label may not match user intent (visually similar ≠ what they want to buy), the crop/detection is wrong, or availability/price filters gut the results post-retrieval. Add a re-ranker on business signals and validate the similarity label against real engagement.

Key takeaways

  • Visual search is representation learning + ANN retrieval — the embedding quality caps everything.
  • Define similarity and mine hard negatives carefully; contrastive/self- supervised training and detect-then-crop handle messy real-world photos.
  • Serve with quantized ANN indexes; a shared image–text space unlocks multimodal "this but in blue" queries.
  • One unified multi-task embedding beats a zoo of specialized ones — in quality and, just as importantly, in operational cost.

References