Design a Real-Time Fraud Detection System
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Design a Real-Time Fraud Detection System

Track

ML System Design

Inspired by

Stripe / Nubank

Scale

billions of tx/day · <100ms · fraud <0.5% · adversarial

Difficulty

Advanced

Fraud detection is the ML interview where the data and the evaluation are the hard parts, not the model. Two forces define it: extreme class imbalance (fraud is a fraction of a percent) and an adaptive adversary who changes behavior the moment you catch them, so the loop's data → evaluation → feedback stages carry the weight. The reference production system here is Stripe Radar, which evaluates over 1,000 characteristics of each payment and decides in under 100 milliseconds (Stripe).

1. Clarify the problem

  • Goal: score each transaction for fraud risk in real time and act — approve, decline, or challenge (step-up auth) — balancing fraud loss against false-positive friction on good customers.
  • Asymmetric costs: a missed fraud loses the transaction value plus a chargeback fee; a false positive blocks a real customer, loses the sale, and erodes trust. These are not symmetric — the threshold is a business decision. Radar's benchmark for the friction side: it falsely blocks about 0.1% of legitimate payments, and even that is a number merchants tune against.
  • Scope: real-time transaction scoring + feedback loop. Related: chargebacks, KYC, account-takeover (mention as adjacent).

Never optimize accuracy

At under 0.5% fraud, predicting "never fraud" is 99.5% accurate and useless. The interview red flag is anyone who reaches for accuracy — use precision/recall at an operating point, AUPRC, and dollar-weighted cost.

Think firstStripe scores every payment inline with over 1,000 features in under 100ms. Which kinds of features does that budget rule out computing at request time — and what does it force you to build instead?Commit to an answer, then reveal

Anything that scans history at request time: "transactions on this card in the last 24h" is a query over billions of rows if computed naively. The budget forces precomputation — streaming aggregations maintained continuously so the request path is a handful of key-value lookups plus a sub-millisecond model call. The architecture is decided by the latency budget before you pick any model.

2. Requirements & scale

DimensionTargetConsequence
Latency< 100 ms inline with checkoutPrecomputed/streaming features; cheap model
VolumeBillions of tx/dayStreaming feature store, not per-request compute
ImbalanceFraud < 0.5%Class weighting, cost-sensitive thresholds, AUPRC
AdversarialFraud shifts weeklyFast retraining + drift monitoring + human review
ExplainabilityReason codes for declinesFeature attributions, often regulatory

For scale calibration: Nubank's risk platform processes about 500 million events per day at 5 million requests per minute across its products (Building Nubank) — this is firmly "streaming infrastructure" territory, not batch scoring.

3. ML problem framing

Cost-sensitive binary classification on a highly imbalanced, non-stationary stream. The decision isn't argmax — it's a threshold (or multiple bands: approve / challenge / decline) chosen to minimize expected dollar cost. Anomaly detection helps for novel patterns, but supervised learning on labeled chargebacks is the core.

What was breaking before? Static rules ("decline if amount > X from new device") are brittle and gamed instantly; a plain classifier ignores the asymmetric costs and the imbalance.

Think firstYour label is “this transaction got a chargeback.” Before building features — what's wrong with that label?Commit to an answer, then reveal

It's late (chargebacks arrive weeks after the transaction, so recent data is unlabeled), noisy (not all fraud is disputed, not all disputes are fraud), and selection-biased (you only ever see chargebacks on transactions you approved). Each flaw needs its own countermeasure — maturation windows, human review as a faster label source, and a bounded randomized-approval slice.

4. Data & features (spine: data)

  • Labels: chargebacks / confirmed fraud — delayed (weeks) and noisy (not all fraud is reported; not all disputes are fraud). Label latency is a first-class problem.
  • Velocity features are the most predictive — counts/sums over short windows: tx_count_5min, distinct_countries_24h, device_age, merchant_decline_rate. These require a streaming feature store answering count/sum/distinct in under 10ms with sub-second freshness, computed by the same pipeline for train and serve.
  • Network features: a processor sees the same card across many businesses — Stripe's core data advantage is that most cards hitting one merchant have already been seen elsewhere on the network, so card- and device-history features exist even for a merchant's first-ever transaction.
  • Graph signals: fraudsters reuse devices, cards, and addresses — a shared-entity graph exposes rings that look clean per-transaction. Grab builds exactly this graph (accounts, devices, payment instruments) to catch professional syndicates running device farms and GPS-spoofing apps, where per-account behavior looks plausible but the shared-hardware structure gives the ring away (Grab Engineering).
Deep dive
Graph-based fraud detection with GNNs
Includes runnable notebook

Message passing on a graph

0.050.950.500.200.800.35
round0
spread0.900
selectednode 4

Each node's colour is its feature value. One step replaces every node's value with the mean of itself and its neighbours — a single GCN-style round. Click a node to highlight its incoming messages; keep stepping and watch all nodes converge to the same colour (over-smoothing).

Message passing propagates risk across the entity graph: an account linked to known-fraud nodes inherits elevated risk its own features wouldn't reveal.

Think firstTabular features, a 100ms inline budget, and regulators want reason codes for every decline. What model family — and why not the deepest one you know?Commit to an answer, then reveal

Gradient-boosted trees are the classic answer: on tabular velocity features they're strong, they score in microseconds, and feature attributions give the required reason codes. But note the frontier has moved — Stripe ran a GBDT-plus-DNN ensemble for years and now serves a pure deep model, because at their data volume the DNN kept improving where XGBoost plateaued. The honest answer is "GBDT until your data size and team can pay for the deep-learning upgrade, and deep models as feature producers in between."

5. Model (spine: hypothesis space + objective)

  • Baseline: gradient-boosted trees (XGBoost) on velocity + account features — the long-time production default for tabular fraud: strong, fast, interpretable.
  • The production frontier: Stripe Radar ran a "wide-and-deep" XGBoost + DNN ensemble, then moved to a DNN-only architecture with ResNeXt-inspired branches — naively dropping XGBoost cost ~1.5 points of recall, so the branch structure had to re-earn it. The payoff: training time fell over 85%, to under two hours, turning retraining from a nightly job into something that runs several times a day (Stripe Engineering). Fast retraining is a modeling feature when the adversary adapts weekly.
  • Add: entity embeddings and GNN risk scores as precomputed features; for attacks with no labels yet, Grab layers an unsupervised graph autoencoder (GraphBEAN) that flags anomalous subgraph structure and has surfaced previously unknown fraud MOs (Grab Engineering).
  • Objective: class-weighted / focal loss, weights set by business cost of each error type, not just inverse frequency.
Think firstA new attack pattern starts at 2am. Even Stripe's two-hour retrain leaves a multi-hour exposure window. What system component covers the gap — and what does it cost you?Commit to an answer, then reveal

A rules layer beside the model: an analyst ships "decline BIN range X from country Y" in minutes, no training data required. The cost is entropy — rules accumulate, overlap, and silently over-block. Nubank made the rules path a first-class product: a declarative policy engine that cut policy deployment from 3–5 weeks to 1–2 days and now runs 300+ policies in production. Treat rules as a managed system with owners and expiry, not a pile of if-statements.

6. Serving & scaling

Inline with checkout under 100ms: fetch streaming features + precomputed graph score, run the model, map the score to an action band, emit reason codes. A rules/policy layer sits alongside the model for known-bad patterns and instant response to emerging attacks — rules ship in minutes while a retrain takes hours. Nubank's version is a dedicated policy engine with a DSL, policy registry, and feature registry, which took policy changes from a 3–5 week engineering task to a 1–2 day analyst task and scaled from ~20 to 300+ live policies (Building Nubank).

Think firstThe model outputs a risk score. Where do you set the decision threshold — and why is 0.5 the wrong answer?Commit to an answer, then reveal

0.5 assumes errors cost the same in both directions; here a missed fraud costs the transaction value while a false decline costs a customer. Put the threshold where the expected dollar cost curve says, and usually not one threshold but bands: approve below, step-up challenge in the middle, decline above. The threshold is a business decision the model merely parameterizes — Stripe exposes exactly this as a merchant-tunable risk-threshold setting.

7. Evaluation (spine: evaluation)

  • Offline: AUPRC (not AUROC — it's misleading when positives are rare), precision/recall @ operating threshold, and dollar-weighted cost curves.
  • Threshold selection: pick the operating point on the cost curve, not a default 0.5 — often multiple thresholds for approve/challenge/decline.
  • Online: monitor approval rate, fraud loss (basis points of volume), and false-positive complaints; A/B new models on a slice with a shadow period first — Stripe shadow-scores with the new model before it takes over decisions.
Deep dive
ROC-AUC vs PR-AUC — why PR-AUC for rare positives
Includes runnable notebook
Think firstOffline AUPRC improved on the holdout, you shipped, and three weeks later fraud losses are climbing. Walk through the two or three most likely explanations.Commit to an answer, then reveal

(1) The adversary moved — your holdout measured the old attack mix; the new model may even have been gamed by probing (card testers watch which attempts survive). (2) Feedback lag fooled you — chargebacks take weeks, so early "wins" were just unlabeled fraud sailing through. (3) Selection bias — the new model approves a different slice, and you have no labels on what the old model used to decline. This is why fraud evaluation is never a one-shot offline exercise.

8. Monitoring & feedback (spine: feedback)

  • Adversarial drift: the adversary adapts, so distributions shift by design. Monitor feature and score drift; keep a human review queue whose decisions become fresh labels far faster than chargebacks.
  • Fast retrain: the loop must close quickly. This is the operational reason Stripe's 85% training-time cut mattered: retraining several times a day shrinks the window between a new attack and a model that has seen it.
  • Feedback bias: you only get labels on transactions you approved (declined ones never charge back) — correct for this selection bias when retraining, including a small, bounded randomized-approval slice in the risky band.

9. Tradeoffs & alternatives

ChoiceAlternativeWhy
GBDT/DNN + graph + rules ensembleSingle monolithic modelRules react in minutes; graph catches rings; retrains take hours
Cost-sensitive threshold bandsSingle 0.5 cutoffAsymmetric costs + step-up auth reduce friction
Streaming feature storeBatch featuresFraud is real-time; stale features miss velocity attacks
Supervised + graph-anomaly hybridPure supervisedUnsupervised graph models catch novel MOs before labels exist
Frequent retrains (hours)Weekly/monthly retrainsThe adversary's iteration speed sets yours

10. Interviewer follow-ups

Labels arrive weeks late as chargebacks. How do you train?

Treat labels as censored: a recent "no chargeback" isn't confirmed-good yet. Use a maturation window before treating transactions as negatives, weight by label confidence, and lean on the human-review queue for faster, cleaner labels on ambiguous cases.

Fraudsters found a new pattern overnight. Model or rules?

Rules — they deploy in minutes and give instant coverage while you gather labels. The model catches up on the next retrain (hours, if you've invested in fast training like Stripe). A mature system runs both: ML for the broad, learned surface; a governed policy engine (Nubank-style) for known-bad and rapid response.

You only see chargebacks on approved transactions. Isn't that biased?

Yes — selection bias. Declined transactions never generate a label, so the model only learns from what it let through. Mitigate with a small randomized-approval slice (bounded, for the riskiest band), and reject-inference techniques when retraining.

Why does a payments processor have an edge over a single merchant's in-house model?

Network data. Radar scores a merchant's first-ever transaction from a card using history accumulated across the whole Stripe network — velocity, device, and dispute history that no single merchant could observe. The lesson generalizes: in fraud, the moat is label and feature coverage, not model architecture.

Key takeaways

  • Fraud is cost-sensitive classification under extreme imbalance and an adaptive adversary — accuracy and AUROC are traps; use AUPRC and dollar cost.
  • Real-time velocity features, network history, and graph signals carry the model; a streaming feature store prevents train-serve skew.
  • The system is a fast loop: human review generates labels, models retrain in hours (Stripe retrains several times a day), and a governed rules layer covers the gap between attack and retrain.

References