Design an Ad Click-Through-Rate Prediction System
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Design an Ad Click-Through-Rate Prediction System

Track

ML System Design

Inspired by

Meta / Google Ads

Scale

billions of predictions/day · <10ms · calibrated probabilities

Difficulty

Advanced

Ad CTR prediction is the highest-stakes ML slot on the internet — the model's output is multiplied by a bid to decide what runs and what an advertiser pays. Two things make it distinct from feed ranking: the prediction must be a calibrated probability (not just a good ranking), and it feeds an auction with real economics. It's also the domain with the best paper trail — Google and Facebook both published unusually frank accounts of their production CTR systems, and this case leans on them throughout. Walk it through the ML project loop with the objective and evaluation stages front and center.

1. Clarify the problem

  • Goal: for each candidate ad, predict P(click) (and P(conversion)) so the auction can rank by expected value bid × pCTR.
  • Why calibration matters: pricing (second-price / VCG) and budget pacing use the absolute probability, not just the order — a miscalibrated model overspends or underspends advertiser budgets.
  • Scope: the prediction model + how it plugs into the auction. Out of scope: the bidding/budget optimizer internals (mention them).

Interview signal

Say "ranking is not enough — the auction needs a calibrated probability" in the first minute. It separates candidates who've built ad systems from those porting a feed-ranking answer.

2. Requirements & scale

DimensionTargetConsequence
Latency< 10 ms for hundreds of candidate adsTiny, heavily-optimized model; feature caching
VolumeBillions of predictions/dayCost per prediction dominates infra spend
CalibrationpCTR ≈ observed CTR across slicesExplicit recalibration + monitoring
FreshnessAdapt to new ads/trends within hoursOnline / frequent incremental training

Back-of-envelope: scoring hundreds of ads in under 10ms means a model that's cheap per item — wide-and-deep / factorization-based, not a giant transformer per ad.

Think firstFacebook measured what happens when a CTR model trained on one day's data is evaluated on each of the following days. What did the curve look like — and what does it force operationally?Commit to an answer, then reveal

Monotonic decay: prediction quality degrades steadily as the gap between training data and serving traffic grows — new ads launch, campaigns end, user behavior shifts. Their conclusion was that data freshness is worth more than most modeling cleverness, pushing retraining to at least daily and motivating online learning. Google went all the way: FTRL-Proximal updates the model per-example, in a streaming loop. A weekly batch retrain is not a smaller version of this system — it's a different, worse system.

3. ML problem framing

Binary probability estimation under extreme class imbalance (CTRs are often 0.1–5%) and huge sparse feature spaces (billions of IDs). Framed as calibrated binary classification with a logloss objective. The ranking that matters is expected value = bid × pCTR × quality, computed in the auction.

What was breaking before? Rule-based / popularity ad selection ignores context and can't price efficiently; a ranker without calibration breaks the auction's economics.

Think firstCTR is around 1%, so 99% of your training rows are negatives. What's the standard fix — and its hidden cost?Commit to an answer, then reveal

Negative downsampling keeps training tractable and balanced, but it distorts the output: a model trained on downsampled data predicts probabilities that are too high. In most systems that would be harmless (ranking survives a monotone distortion); here the auction prices on the absolute number, so you must analytically correct or recalibrate after downsampling.

4. Data & features (spine: data)

  • Labels: impressions → click (and click → conversion, delayed). Delayed feedback is a real problem: a "no click yet" label may flip later.
  • Features: user, ad (creative embedding, advertiser, category), context (placement, device, time), and dense cross features (user × ad category). ID features are embedded; the embedding tables dominate model size.
  • Imbalance: negative downsampling with calibration correction to recover true probabilities after resampling — Facebook's recipe: downsample negatives by rate ww, then map the model output back with q=p/(p+(1−p)/w)q = p / (p + (1-p)/w) (He et al., ADKDD 2014).
  • Freshness: the same paper showed model quality decays day over day when training lags serving, so labels flow into (at least) daily retrains, and the feature pipeline is streaming, not nightly.
Think firstYour feature space is billions of sparse IDs (every user × ad × placement cross). One weight per feature can't fit in serving memory. What are your levers before you reach for a smaller model?Commit to an answer, then reveal

Three, all from Google's production playbook: hashing IDs into a bounded space, an L1-inducing online update (FTRL-Proximal) so most weights are exactly zero and never stored, and probabilistic feature inclusion — only admit a feature once it has appeared enough times to plausibly matter. The model stays expressive over billions of possible features while materializing only the tiny fraction that earns its memory.

5. Model (spine: hypothesis space + objective + optimization)

  • Baseline: logistic regression with hashed cross features, trained online — Google's ads system ran on exactly this, with FTRL-Proximal per-coordinate updates over billions of coefficients (McMahan et al., KDD 2013). Still a shockingly strong, cheap baseline.
  • Feature learning: Facebook's classic upgrade kept the LR but fed it GBDT leaf indices as learned categorical features — trees as automatic feature crossers — beating either model alone by >3% in normalized entropy (He et al., ADKDD 2014).
  • Production today: a Wide & Deep / DLRM-family network — a wide linear side that memorizes proven crosses plus a deep embedding side that generalizes to unseen ones, the split Google introduced for Play recommendations (Cheng et al., 2016); at Meta scale the sparse embedding tables dominate model size (DLRM). Objective: logloss (a proper scoring rule → calibrated by construction, if not distorted).
  • Recalibration: Platt scaling / isotonic regression on a held-out slice to correct residual miscalibration (and the downsampling bias).
Deep dive
Calibration — Platt scaling & isotonic regression
Includes runnable notebook

Calibration: reliability diagram & temperature scaling

confidenceaccuracy0.50.50.750.7511
avg confidence0.827
avg accuracy0.710
ECE0.120

Each bar groups predictions by confidence; its height is the actual accuracy in that bin. Perfect calibration lies on the diagonal (confidence = accuracy). The model is overconfident at temperature 1 (bars below the line). Raise the temperature to soften the probabilities; the Expected Calibration Error (ECE) bottoms out when the bars meet the diagonal.

A reliability diagram like this is the artifact you monitor: points on the diagonal mean predicted probabilities match observed click rates.

Think firstThe model emits pCTR = 0.03 for an ad. What consumes that number next, and what does that imply about the model's real job?Commit to an answer, then reveal

The auction: ads are ranked by bid × pCTR, and the price the winner pays is derived from the runner-up's expected value. So the model isn't producing a relevance score — it's producing one factor in a price. Overestimate pCTR by 2× and you overcharge the advertiser by 2×; the model's real job is honest probabilities, not just correct ordering.

6. Serving & the auction

Prediction feeds the auction: ads are ranked by bid × pCTR (× quality), and the winner pays a price derived from the runner-up (second-price / GSP / VCG). Budget pacing throttles delivery so advertisers spend evenly.

Deep dive
Ad auctions — second-price, GSP, VCG
Includes runnable notebook

Serving: feature-fetch from a low-latency store, batch-score all candidate ads, apply pacing/floor rules, run the auction — all under 10ms.

Think firstWhich two offline numbers must you track — and how can one improve while the other silently loses money?Commit to an answer, then reveal

AUC (ranking) and calibration (ECE / reliability curves). They're independent: a new model can order ads better and systematically overpredict probabilities, and since price = bid × pCTR, the calibration regression misprices every auction while your AUC dashboard glows green. Track both, per segment.

7. Evaluation (spine: evaluation)

  • Offline: AUC / logloss for ranking + calibration metrics (ECE, reliability curves). AUC can improve while calibration degrades — track both.
  • Online: A/B on revenue and advertiser ROI, with guardrails on user experience (ad load, relevance) and calibration by segment.
  • Counterfactual: replay/off-policy estimation before spending live traffic.
Think firstAggregate calibration is perfect: mean pCTR equals observed CTR to the third decimal. Why can the system still be losing money every hour?Commit to an answer, then reveal

Because calibration errors can cancel in aggregate: overpredicting on video placements while underpredicting on mobile ones nets to zero overall, yet every video auction is overpriced and every mobile one underdelivers — advertisers on each side notice even when your dashboard doesn't. Calibration is only meaningful per slice (placement, advertiser vertical, device, ad age); the aggregate number is a vanity metric. This is why calibration monitoring, not AUC monitoring, is the pager-worthy alert.

8. Monitoring & feedback (spine: feedback)

  • Calibration drift by slice (advertiser, placement) is the alarm that matters most — it silently distorts pricing.
  • Delayed conversions: a conversion can land up to a month after the click, so recent "negatives" are really censored labels — model the delay distribution explicitly rather than mislabeling them (Chapelle, KDD 2014), and backfill labels as conversions arrive. The loop closes into data.

9. Tradeoffs & alternatives

ChoiceAlternativeWhy
Calibrated prob + auctionRank-only scorePricing/pacing need absolute probabilities
DLRM (embeddings + crosses)Deep transformer per adLatency/cost: the model runs billions of times/day
Negative downsampling + recalibrationTrain on all negativesCheaper; recalibration recovers true probabilities
Logloss objectivePure AUC optimizationLogloss is a proper score → calibrated; AUC ignores calibration

10. Interviewer follow-ups

AUC went up but revenue dropped. What happened?

Almost always calibration: a higher-AUC model that's miscalibrated misprices the auction — over/under-bidding relative to true value. Check ECE and per-slice reliability; recalibrate with isotonic regression before shipping.

How do you handle delayed conversions in the label?

Model the delay: use a delayed-feedback model that treats "no conversion yet" as censored rather than negative, or wait for a fixed attribution window and backfill. Naively labeling recent impressions as negatives biases pCVR downward.

A brand-new ad has no history — how is it priced?

Content/advertiser-level features and category priors give an initial pCTR; an exploration budget grants early impressions to gather signal (explore/exploit). Cold start in ads is expensive, so exploration is bounded and tracked.

Key takeaways

  • Ad CTR prediction must produce a calibrated probability, because it feeds an auction that prices on the absolute number.
  • Latency and cost force small, embedding-heavy models (DLRM-style), not giant networks — it runs billions of times a day.
  • Data freshness beats model cleverness: day-old training data measurably degrades predictions, which is why the field invented online learning (FTRL) and daily GBDT+LR refresh cycles.
  • Track AUC and calibration separately, and calibration per slice; a ranking win with a calibration regression loses money.

References