Thinking in ProbabilitiesOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Thinking in Probabilities

Machine learning is the business of making decisions under uncertainty — a model never knows the label, it believes it with some strength. Probability is the language those beliefs are written in. Two stages of the ML project loop lean hardest on it: the objective (almost every loss is a negative log-likelihood) and evaluation (every confidence interval and p-value). Before meeting the famous distributions, it pays to nail down the small set of words everything else is built from.

Deep dive
The ML Project Loop
Includes runnable notebook

Sample spaces and events

A sample space Ω\Omega is the set of everything that could happen in one run of an experiment:

  • One coin flip: Ω={H,T}\Omega = \{H, T\}
  • One die roll: Ω={1,2,3,4,5,6}\Omega = \{1, 2, 3, 4, 5, 6\}
  • Tomorrow's temperature: Ω=R\Omega = \mathbb{R} (uncountably many outcomes!)

An event is any subset of outcomes you might care about: "the roll is even" is the event A={2,4,6}A = \{2, 4, 6\}. A probability function PP assigns each event a number, subject to three rules (the Kolmogorov axioms):

  1. P(A)≥0P(A) \ge 0 — no negative beliefs
  2. P(Ω)=1P(\Omega) = 1 — something happens
  3. If AA and BB can't both happen, P(A∪B)=P(A)+P(B)P(A \cup B) = P(A) + P(B) — probabilities of exclusive events add

That's the entire rulebook. Everything else (Bayes' theorem, likelihoods, expectation) is a consequence.

Two readings of the same number

What does P(rain)=0.3P(\text{rain}) = 0.3 mean? The frequentist reading: in many repeats of days like this one, 30% are rainy. The Bayesian reading: 0.3 is a degree of belief, which new evidence should update. The axioms above don't care which story you tell — both readings obey the same math, and ML uses each where it fits (Bayesian inference leans on the second).

Conditional probability: the comma and the bar

Two notations carry most of probability's meaning:

P(A,B)=both happenP(A∣B)=A happens, given B didP(A, B) = \text{both happen} \qquad P(A \mid B) = \text{$A$ happens, given $B$ did}

They're linked by the product rule, arguably the most-used identity in ML:

P(A,B)=P(A∣B) P(B)P(A, B) = P(A \mid B)\, P(B)

Conditioning is zooming in: P(A∣B)P(A \mid B) throws away every world where BB didn't happen and renormalizes what's left. When the zoom changes nothing (P(A∣B)=P(A)P(A \mid B) = P(A)), the events are independent, and the product rule collapses to P(A,B)=P(A)P(B)P(A,B) = P(A)P(B). (Naive Bayes builds an entire classifier from pretending this collapse holds.)

Both of them, as areas of one square

The comma and the bar are different questions

01P(B) = 0.010A¬AA and Barea = 0.0099A but not Barea = 0.0495shaded totalP(A) = 0.0594teal ÷ shadedP(B|A) = 0.1667
P(A, B) — the comma0.0099
P(A | B)0.9900
P(B | A)0.1667
ratio between the two bars5.94×
P(A)0.0594

One unit square, drawn to scale. Its width splits by B (left column) and ¬B; inside each column the height splits by A. P(A,B) is the teal area. P(A|B) is that area as a fraction of its own column; P(B|A) is the same area as a fraction of everything shaded. The comma is symmetric — both bars are not, and they divide by different denominators.

Every quantity above is a region of the same unit square, which is the cheapest way to keep them apart. The width splits by BB; inside each column the height splits by AA. Then P(A,B)P(A,B) is an area, and each conditional is that area divided by a different denominator — P(A∣B)P(A\mid B) by its own column, P(B∣A)P(B\mid A) by everything shaded.

The comma is symmetric. The bars are not, and the default preset shows how badly. Base rate P(B)=0.01P(B) = 0.01, sensitivity P(A∣B)=0.99P(A\mid B) = 0.99, false positives P(A∣¬B)=0.05P(A\mid\lnot B) = 0.05:

P(A)=0.01(0.99)+0.99(0.05)=0.0594P(B∣A)=0.00990.0594=0.1667P(A) = 0.01(0.99) + 0.99(0.05) = 0.0594 \qquad P(B\mid A) = \frac{0.0099}{0.0594} = 0.1667

A test that is right 99% of the time when you are ill leaves you 16.7% likely to be ill when it fires — a factor of 5.9 between the two conditionals. You do not need Bayes' rule to see why: the square is drawn to scale, and the sliver of true positives is simply dwarfed by a false-positive band that starts out 99× wider. Swapping P(A∣B)P(A\mid B) for P(B∣A)P(B\mid A) is the single most common probability error in applied work, and it is this picture read in the wrong direction.

Drag P(B)P(B) up and the two bars converge; press independent and the split stops mattering at all — P(A∣B)=P(A∣¬B)=P(A)P(A\mid B) = P(A\mid \lnot B) = P(A), and the vertical line no longer changes any height.

Random variables

A random variable XX attaches a number to each outcome — the die's face value, a pixel's brightness, tomorrow's temperature. This is the bridge from raw outcomes to arithmetic: once outcomes are numbers, you can average them, weight them, and feed them to models.

  • Discrete XX takes countable values, described by a probability mass function: P(X=k)P(X = k)
  • Continuous XX takes values in a continuum, described by a probability density function, where only areas are probabilities: P(a≤X≤b)=∫abp(x) dxP(a \le X \le b) = \int_a^b p(x)\,dx

The next lesson makes this concrete with the four distributions ML reaches for daily.

Deep dive: why a density can exceed 1

A common trip-up: a PDF value is not a probability. A uniform on [0,0.5][0, 0.5] has density p(x)=2p(x) = 2 everywhere on that interval — perfectly legal, because only the area must equal 1, and 2×0.5=12 \times 0.5 = 1. Probabilities come from integrating the density over a range; the value at a single point can be arbitrarily large (and the probability of any exact single value is 0). Heights trade off against widths so the total area stays 1. You can see this live in the distribution visualization in the next lesson.

Expectation: the long-run average

The expectation of a random variable is its probability-weighted average:

E[X]=∑kk P(X=k)orE[X]=∫x p(x) dx\mathbb{E}[X] = \sum_k k \, P(X = k) \qquad \text{or} \qquad \mathbb{E}[X] = \int x \, p(x) \, dx

Roll a fair die forever and your average roll converges to E[X]=3.5\mathbb{E}[X] = 3.5 — a value the die never actually shows. Expectation is the center of mass of belief, not a prediction of any single outcome.

Variance measures spread around that center: Var(X)=E[(X−E[X])2]\mathrm{Var}(X) = \mathbb{E}[(X - \mathbb{E}[X])^2].

You can watch the law of large numbers pull a running average onto the expectation in a few lines:

Python
import numpy as np

rng = np.random.default_rng(0)
rolls = rng.integers(1, 7, size=10_000)

running_mean = np.cumsum(rolls) / np.arange(1, rolls.size + 1)
print(running_mean[[9, 99, 999, 9999 - 1]])
# → drifts toward 3.5 as the sample grows: e.g. [3.1, 3.38, 3.46, 3.50]

Why ML cares so much about 𝔼

Almost every training objective is an expectation: the loss you minimize is the expected loss over the data distribution, approximated by averaging over your training set. Gradient descent on a mini-batch works precisely because the mini-batch average is an unbiased estimate of that expectation.

Exercise

A fair six-sided die has expectation E[X] = 3.5. What does that number mean?

Need a hint?

Can a die ever show 3.5?

Key takeaways

  • Probability assigns numbers to events under three axioms: non-negative, sums to 1, additive over exclusive events
  • The product rule P(A,B)=P(A∣B)P(B)P(A,B) = P(A \mid B)P(B) links joint and conditional probability; independence means conditioning changes nothing
  • Random variables turn outcomes into numbers; discrete ones have mass functions, continuous ones have densities (areas = probabilities)
  • Expectation is the probability-weighted average — the quantity nearly every ML loss function is secretly approximating

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedGo deeper
Thinking in ProbabilitiesProbability DistributionsBayesian InferenceMaximum Likelihood Estima…The ML Project LoopProbability Distributions…