Thinking in Probabilities
Machine learning is the business of making decisions under uncertainty — a model never knows the label, it believes it with some strength. Probability is the language those beliefs are written in. Two stages of the ML project loop lean hardest on it: the objective (almost every loss is a negative log-likelihood) and evaluation (every confidence interval and p-value). Before meeting the famous distributions, it pays to nail down the small set of words everything else is built from.
Sample spaces and events
A sample space is the set of everything that could happen in one run of an experiment:
- One coin flip:
- One die roll:
- Tomorrow's temperature: (uncountably many outcomes!)
An event is any subset of outcomes you might care about: "the roll is even" is the event . A probability function assigns each event a number, subject to three rules (the Kolmogorov axioms):
- — no negative beliefs
- — something happens
- If and can't both happen, — probabilities of exclusive events add
That's the entire rulebook. Everything else (Bayes' theorem, likelihoods, expectation) is a consequence.
Two readings of the same number
What does mean? The frequentist reading: in many repeats of days like this one, 30% are rainy. The Bayesian reading: 0.3 is a degree of belief, which new evidence should update. The axioms above don't care which story you tell — both readings obey the same math, and ML uses each where it fits (Bayesian inference leans on the second).
Conditional probability: the comma and the bar
Two notations carry most of probability's meaning:
They're linked by the product rule, arguably the most-used identity in ML:
Conditioning is zooming in: throws away every world where didn't happen and renormalizes what's left. When the zoom changes nothing (), the events are independent, and the product rule collapses to . (Naive Bayes builds an entire classifier from pretending this collapse holds.)
Both of them, as areas of one square
The comma and the bar are different questions
One unit square, drawn to scale. Its width splits by B (left column) and ¬B; inside each column the height splits by A. P(A,B) is the teal area. P(A|B) is that area as a fraction of its own column; P(B|A) is the same area as a fraction of everything shaded. The comma is symmetric — both bars are not, and they divide by different denominators.
Every quantity above is a region of the same unit square, which is the cheapest way to keep them apart. The width splits by ; inside each column the height splits by . Then is an area, and each conditional is that area divided by a different denominator — by its own column, by everything shaded.
The comma is symmetric. The bars are not, and the default preset shows how badly. Base rate , sensitivity , false positives :
A test that is right 99% of the time when you are ill leaves you 16.7% likely to be ill when it fires — a factor of 5.9 between the two conditionals. You do not need Bayes' rule to see why: the square is drawn to scale, and the sliver of true positives is simply dwarfed by a false-positive band that starts out 99× wider. Swapping for is the single most common probability error in applied work, and it is this picture read in the wrong direction.
Drag up and the two bars converge; press independent and the split stops mattering at all — , and the vertical line no longer changes any height.
Random variables
A random variable attaches a number to each outcome — the die's face value, a pixel's brightness, tomorrow's temperature. This is the bridge from raw outcomes to arithmetic: once outcomes are numbers, you can average them, weight them, and feed them to models.
- Discrete takes countable values, described by a probability mass function:
- Continuous takes values in a continuum, described by a probability density function, where only areas are probabilities:
The next lesson makes this concrete with the four distributions ML reaches for daily.
Deep dive: why a density can exceed 1
A common trip-up: a PDF value is not a probability. A uniform on has density everywhere on that interval — perfectly legal, because only the area must equal 1, and . Probabilities come from integrating the density over a range; the value at a single point can be arbitrarily large (and the probability of any exact single value is 0). Heights trade off against widths so the total area stays 1. You can see this live in the distribution visualization in the next lesson.
Expectation: the long-run average
The expectation of a random variable is its probability-weighted average:
Roll a fair die forever and your average roll converges to — a value the die never actually shows. Expectation is the center of mass of belief, not a prediction of any single outcome.
Variance measures spread around that center: .
You can watch the law of large numbers pull a running average onto the expectation in a few lines:
import numpy as np
rng = np.random.default_rng(0)
rolls = rng.integers(1, 7, size=10_000)
running_mean = np.cumsum(rolls) / np.arange(1, rolls.size + 1)
print(running_mean[[9, 99, 999, 9999 - 1]])
# → drifts toward 3.5 as the sample grows: e.g. [3.1, 3.38, 3.46, 3.50]
Why ML cares so much about 𝔼
Almost every training objective is an expectation: the loss you minimize is the expected loss over the data distribution, approximated by averaging over your training set. Gradient descent on a mini-batch works precisely because the mini-batch average is an unbiased estimate of that expectation.
Exercise
A fair six-sided die has expectation E[X] = 3.5. What does that number mean?
Need a hint?
Can a die ever show 3.5?
Key takeaways
- Probability assigns numbers to events under three axioms: non-negative, sums to 1, additive over exclusive events
- The product rule links joint and conditional probability; independence means conditioning changes nothing
- Random variables turn outcomes into numbers; discrete ones have mass functions, continuous ones have densities (areas = probabilities)
- Expectation is the probability-weighted average — the quantity nearly every ML loss function is secretly approximating
Related concepts
- Probability Distributions: the named shapes (Gaussian, uniform, Bernoulli, Poisson) that package these definitions for daily use
- Maximum Likelihood Estimation: turning "probability of data given parameters" around into a fitting principle
- Bayesian Inference: the product rule run in reverse: updating beliefs from evidence