Privacy & Federated LearningOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Privacy & Federated Learning

Models trained on personal data (health records, messages, keystrokes, location) can leak that data: they memorize training examples, and attackers can extract them or infer who was in the training set. As ML touches more sensitive data (and regulation tightens), training that provably protects individuals has become a first-class engineering concern. This lesson covers the two pillars: differential privacy (a rigorous guarantee) and federated learning (keep the data on the device).

Real-world examples: mobile keyboards learning next-word prediction without uploading your texts, hospitals jointly training a model without sharing patient records, the US Census applying differential privacy to published statistics, and DP-trained foundation models.

Models leak training data

Neural networks memorize. Documented attacks:

  • Membership inference: given a data point, determine whether it was in the training set (a privacy violation by itself for, say, a medical dataset).
  • Training-data extraction: prompt a large model to regurgitate verbatim training examples (names, phone numbers, secrets).

Anonymization alone is famously weak: "de-identified" datasets are routinely re-identified by linking to side information. We need a mathematical guarantee, not a hope.

Differential privacy

Differential privacy (DP) is that guarantee. Intuitively: the model's output should be almost the same whether or not any single individual's data was included, so no one's participation can be detected, and nothing specific to them can be learned. Formally, a randomized algorithm MM is ε\varepsilon-differentially private if for any two datasets D,D′D, D' differing in one person's record, and any outcome SS:

Pr⁡[M(D)∈S]≤eε Pr⁡[M(D′)∈S]\Pr[M(D) \in S] \le e^{\varepsilon}\,\Pr[M(D') \in S]
  • ε\varepsilon (the privacy budget): smaller = more private. ε→0\varepsilon \to 0 means the two cases are indistinguishable (max privacy, useless model); large ε\varepsilon means weak privacy. Typical deployments target ε\varepsilon in roughly 11–1010.
  • The guarantee is achieved by adding calibrated random noise — to the data, the gradients, or the output.

The definition above is pure ε\varepsilon-DP, achieved exactly by the Laplace mechanism (noise ∼Lap(0,Δf/ε)\sim \text{Lap}(0, \Delta f / \varepsilon) for a query with ℓ1\ell_1-sensitivity Δf\Delta f). In practice, DP-SGD uses Gaussian noise, which satisfies the relaxed (ε,δ)(\varepsilon, \delta)-DP guarantee: with small but nonzero δ\delta (typically δ≪1/n\delta \ll 1/n), the multiplicative bound may fail with probability at most δ\delta. The (ε,δ)(\varepsilon, \delta) variant is written:

Pr⁡[M(D)∈S]≤eε Pr⁡[M(D′)∈S]+δ.\Pr[M(D) \in S] \le e^{\varepsilon}\,\Pr[M(D') \in S] + \delta.

Gaussian noise is preferred in practice because it leads to tighter privacy accounting across many gradient steps (using the Rényi or moments accountant), at the cost of introducing the δ\delta slack.

DP is a property of the algorithm, not the data

You can't look at a released model and check if it's 'private.' DP is a guarantee about the training procedure: how much any one record could have influenced the result. That's what makes it robust — it holds against any attacker with any side information, unlike ad-hoc anonymization.

DP-SGD: differentially private training

The standard way to train a deep network with DP is DP-SGD, which modifies normal SGD in two steps each update:

  1. Clip per-example gradients to a maximum norm CC — bounding how much any single example can move the model (its "sensitivity").
  2. Add Gaussian noise to the summed gradients before the step — masking any individual's contribution.
g~=1B(∑iclip(gi,C)+N(0,σ2C2I))\tilde{g} = \frac{1}{B}\left(\sum_{i} \text{clip}(g_i, C) + \mathcal{N}(0, \sigma^2 C^2 \mathbf{I})\right)

The clip-then-noise pattern is the heart of it. A privacy accountant tracks the total ε\varepsilon spent across all training steps (privacy budget is consumed cumulatively).

The privacy–utility trade-off is real

Clipping and noise cost accuracy — more privacy (smaller ε\varepsilon, more noise) means a worse model. DP-SGD also disproportionately hurts underrepresented groups, a fairness concern. There is no free privacy: you are always buying it with utility.

Which of the two operations actually costs you

Which half of DP-SGD actually costs you

The clipstep 1 of 4
The minority's gradients are larger, not rarer
Per-example gradient norms after training, split by group. The minority’s median is 3.76× the majority’s — because the model fits them worse, so their gradients never shrink.

That single fact drives everything that follows. Clipping is a ceiling on gradient norm, and this group is the one sitting above it.

Move the clip norm C and watch the two clipped-percentage figures separate.

whose gradients are bigno noise yet · clipping only
clip at C = 1.0majority (1066)minority (134)0.00.30.60.91.2per-example gradient norm (each group scaled to its own peak; last bin is everything above)
median gradient norm

majority 0.096 · minority 0.362
ratio 3.76×

clipped at C = 1.0

majority 3% · minority 16%
4.6× the rate

majority gradientsminority gradientsclip norm

1,200 examples, 10% from a minority group whose labelling rule differs. The model carries group-interaction features, so the minority rule is fully representable and only its sample count limits it. Full-batch DP gradient descent, 80 steps; because the batch is not subsampled the RDP accountant shown is exact. Every accuracy and gradient norm below is trained and measured here.

The disparate impact is usually explained as noise drowning out rarer gradients. Simulate it and both halves of that explanation come apart.

The minority's gradients are larger, not rarer. The model fits them worse, so their gradients never shrink: after training, the median per-example gradient norm is 3.76× the majority's. Clipping is a ceiling on gradient norm, so this is exactly the group sitting above it — at C=1C = 1, 16% of minority examples are clipped against 3% of majority, five times the rate, every step.

And it is the clipping, not the noise. With zero noise (no privacy guarantee purchased at all) tightening CC from 20 to 0.2 leaves majority accuracy flat at 0.95 while minority accuracy falls from 0.88 to chance. Holding C=1C = 1 and sweeping the noise instead, the minority barely moves from σ=0\sigma = 0 through σ=32\sigma = 32 (ε≈1.4\varepsilon \approx 1.4). Averaged over a large batch the noise is amortised across every example; the per-example clip is not.

That changes which knob to reach for. Spending more ε\varepsilon buys back almost nothing here — the damage was done by the operation that carries no privacy guarantee. Per-group or adaptive clip norms target the mechanism that is actually responsible. This matches the finding in Bagdasaryan et al., Differential Privacy Has Disparate Impact on Model Accuracy (2019), which isolates clipping as the dominant cause.

The harm also scales sharply with representation: at a 5% minority share the accuracy gap is about 0.52; at 20% it is 0.04; at 50% it is essentially zero. Same algorithm, same ε\varepsilon, same clip norm — the damage is set by how many examples the group has to spare.

Federated learning

Federated learning (FL) attacks privacy from a different angle: don't move the data at all. Instead of uploading raw data to a central server, the model goes to the data:

Text
1. Server sends the current model to many clients (phones, hospitals).
2. Each client trains locally on its own private data.
3. Clients send back only model UPDATES (gradients/weights), never raw data.
4. Server aggregates the updates (FedAvg = weighted average) into a new model. Repeat.

Federated Averaging (FedAvg) is the core algorithm: average the clients' locally-updated weights, weighted by their data counts. The raw data never leaves the device.

FL's own challenges: clients have non-IID data (your phone's data isn't representative of everyone's), unreliable/intermittent participation, communication cost, and — importantly — the updates themselves can leak (gradients can be inverted to partially reconstruct inputs).

The techniques compose

FL and DP are complementary and usually combined: federated learning keeps raw data local, secure aggregation ensures the server sees only the summed update (not any individual's), and differential privacy adds noise so even the aggregate can't be inverted to a person. Defense in depth — each layer closes a different leak.

Exercise

Differential privacy guarantees that the model's output is nearly the same whether or not any one individual's data is included, tuned by ε. What does a SMALLER ε mean?

Need a hint?

ε bounds how distinguishable the two cases are.

Exercise

DP-SGD modifies normal SGD with two steps per update. What are they, and what is the clipping for?

Need a hint?

Bound each example's influence, then mask it.

Exercise

In federated learning with FedAvg, what is sent from the clients back to the server?

Need a hint?

What stays on the device, and what leaves it?

Common mistakes

  1. Trusting anonymization. De-identified data is re-identifiable; use DP for an actual guarantee.
  2. Treating ε as a label, not a budget. Privacy is spent cumulatively across queries/epochs; an accountant must track the total.
  3. Assuming federated = private. Raw data stays local, but gradient updates can leak — add secure aggregation and/or DP.
  4. Ignoring the utility (and fairness) cost. DP noise lowers accuracy and hits rare groups hardest; budget it deliberately.

Key takeaways

  • Models memorize and leak training data (membership inference, extraction); anonymization is not a real defense.
  • Differential privacy guarantees that any single individual barely affects the output, tuned by the privacy budget ε\varepsilon (smaller = more private) via calibrated noise.
  • DP-SGD trains privately by clipping per-example gradients then adding Gaussian noise, with an accountant tracking total ε\varepsilon — at a real accuracy cost.
  • Federated learning keeps raw data on-device and shares only model updates (FedAvg); combine it with secure aggregation + DP because updates can still leak.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Privacy & Federated LearningLinear & Logistic Regress…Neural NetworksModel Evaluation & Valida…Responsible AI & the Huma…MLOps Infrastructure & Or…Gradient Descent Variants