Fraud Detection at ScaleOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Fraud Detection at Scale

Anomaly detection identifies statistical outliers. Fraud detection is harder: the adversary actively mimics normal behavior, so the fraud distribution constantly shifts. A transaction at 2 AM from a new device in a foreign country might be fraud, or it might be a legitimate traveler. Getting the decision wrong in either direction has asymmetric costs: a missed fraud loses money; a false alarm blocks a genuine customer and erodes trust.

Real-world examples: Nubank (scaling ML fraud defense to 35 M+ customers), Stripe (payment fraud at millions of merchants), Uber (driver/rider identity fraud, promo abuse), Binance (exchange-level fraud & wash trading), Feedzai (bank transaction screening), Wayfair (return fraud), DoorDash (fraudulent orders).

The class-imbalance problem

Fraud rates are typically 0.01%–0.5% of all transactions. A model that predicts "not fraud" for every transaction has 99.5%+ accuracy, and zero utility.

Metrics that matter

Never use accuracy. Instead:

  • Precision@K / Recall@K: given a ranked list of most-suspicious transactions, what fraction are truly fraud (precision) and what fraction of all fraud did you catch (recall)?
  • AUPRC (Area Under Precision-Recall Curve): better than AUROC when the positive class is rare.
  • F-beta score: Fβ=(1+β2)P⋅Rβ2P+RF_\beta = (1+\beta^2)\frac{P \cdot R}{\beta^2 P + R}; set β>1\beta > 1 (e.g., β=2\beta = 2) to weight recall over precision when missing fraud is more costly than a false alarm.

Resampling and re-weighting

TechniqueMethodTrade-off
Random undersamplingRemove majority class samplesFast; discards real data
SMOTESynthesize minority class examplesBetter recall; risk of overfitting to synthetic border cases
Class weightssample_weight = 1/class_freq in lossSimple; no data distortion

In production, class weighting is preferred: it's trivially applied to any training objective and avoids the instability of SMOTE at scale. At Stripe, models are trained with class weights proportional to the business cost of each error type, not just the inverse frequency.

Feature engineering for fraud

The most predictive features are velocity features: how much has this entity done in a short window?

Code
# Real-time velocity features for a transaction
user_tx_count_5min     = count(user transactions, last 5 min)
user_amount_1hr        = sum(transaction amount, last 1 hour)  
card_countries_24hr    = distinct(countries seen for this card, last 24hr)
merchant_decline_rate  = declines / attempts for this merchant, last 7 days
device_first_seen_days = age of device fingerprint

These require a real-time feature store that can answer count/sum/distinct queries at < 10 ms with < 1 s freshness. Nubank uses Apache Flink for streaming feature computation; the same Flink job that computes features for training also serves them at inference time, eliminating training-serving skew.

Entity embeddings for fraud

Entities like merchants, devices, and IP addresses have hundreds of attributes. Rather than engineering every attribute manually, train entity embeddings jointly with the fraud model:

  • Each entity (card BIN, merchant category, IP/24) gets a dd-dimensional embedding learned from transaction history.
  • The embeddings capture latent behavioral profiles: merchant categories that are fraud hotspots cluster in embedding space.
  • Unseen entities fall back to the average embedding for their category.

Graph-based fraud detection

Fraud doesn't happen in isolation — fraudsters often reuse phone numbers, device IDs, billing addresses, and IP addresses across multiple accounts. This creates a bipartite graph connecting accounts to shared attributes.

The homophily observation

In fraud graphs, bad actors cluster: account nodes connected through shared device IDs are much more likely to be co-fraudulent than accounts connected only through shared names. This is the homophily property that GNNs exploit.

y^v=f ⁣(hv,  AGGREGATE ⁣({hu:u∈N(v)}))\hat{y}_v = f\!\left(\mathbf{h}_v,\; \text{AGGREGATE}\!\left(\left\{\mathbf{h}_u : u \in \mathcal{N}(v)\right\}\right)\right)

A GNN-based fraud model at Uber aggregates features from the neighborhood of each account node. An account connected to 3 known-fraud devices gets a high risk score even if its own features look benign. This catches synthetic identity fraud (a new clean-looking account that shares a phone number with a known fraudster).

Deep dive
Graph-Based Fraud Detection with GNNs
Includes runnable notebook

Graph updates must be near-real-time

A graph edge "account X shares device with account Y (known fraud)" is useless if it's only computed in a nightly batch job. Fraudsters operate in burst campaigns lasting minutes. Uber and Nubank maintain incremental graph updates: when account Y is labeled fraud, all neighboring nodes are immediately re-scored.

The adversarial dimension

Unlike most ML problems, fraud detection faces an adaptive adversary — the fraud population evolves in response to your model. After a model deployment:

  1. Fraud that matches the old pattern is blocked.
  2. Fraudsters adapt: change device fingerprints, use mule accounts, run smaller transactions.
  3. The new fraud is under-represented in training data.
  4. Model performance degrades silently.

Defenses

Concept drift monitoring: track the statistical distribution of flagged fraud features weekly. A shift in the distribution of fraud country-of-origin or transaction amount is an early warning that the fraud pattern has evolved.

Champion-challenger online deployment: run a new candidate model alongside the production model on a traffic split, comparing Recall@FPR on the same live transactions. When challenger > champion by a statistically significant margin, rotate.

Feedback loop hardening: never use the model's own fraud flags as the only source of ground-truth labels. Maintain a manual review queue for high-uncertainty cases; human reviewers provide unbiased signal on the true distribution.

Real-time scoring architecture

Code
Transaction event (< 100 ms budget)
      ↓
Real-time feature computation (Flink, Redis)
  ├─ Velocity features (last-N-minutes counts)
  ├─ Entity embeddings (pre-computed, cached)
  └─ Graph neighbor scores (incremental GNN)
      ↓
Risk scoring model (GBDT or neural)
      ↓
Decision thresholds:
  ├─ score > 0.90 → auto-decline
  ├─ 0.60 < score < 0.90 → step-up auth (SMS OTP)
  └─ score < 0.60 → approve
      ↓
Outcome logged → retraining pipeline

Binance's fraud stack processes millions of transactions per second with a p95 latency of < 40 ms for the full scoring pipeline.

Exercise

A fraud detection model achieves 99.8% accuracy on a dataset where only 0.1% of transactions are fraud. Should you trust this accuracy number?

Need a hint?

What would a trivial baseline that predicts 'not fraud' for everything score?

Exercise

Which feature type is most predictive for real-time payment fraud detection and requires a streaming feature store?

Need a hint?

Think about what changes in the minutes before a fraud event.

Exercise

Graph-based fraud detection uses GNNs over an entity-sharing graph. What specific fraud pattern does this catch that feature-only models miss?

Need a hint?

Think about what it means for two accounts to share a device ID.

Key takeaways

  • Standard accuracy is meaningless at 0.1% fraud rate; use AUPRC, F-beta (β > 1), and Recall@FPR as evaluation metrics.
  • Velocity features (count/sum in rolling time windows) are the highest-signal fraud features; they require a real-time feature store.
  • Graph-based detection (GNNs over entity-sharing graphs) catches synthetic identity fraud that evades feature-only models.
  • Fraud is adversarial: the distribution shifts after deployment; champion-challenger comparison and drift monitoring are mandatory.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Fraud Detection at ScaleLinear & Logistic Regress…Neural NetworksModel Evaluation & Valida…Anomaly & Outlier Detecti…Graph Convolutions & Grap…Feature StoresContent Moderation at Sca…Graph-Based Fraud Detecti…