Probability Distributions
Machine learning is fundamentally about uncertainty — uncertain data, uncertain labels, uncertain predictions. Probability theory gives us the language to reason precisely about uncertainty. Most ML models are probability distributions in disguise.
Real-world examples: Classification models output probability distributions over classes; generative models (VAEs, diffusion) model data as distributions; A/B testing uses hypothesis tests grounded in statistical distributions.
Random variables and distributions
Recall from Thinking in Probabilities: a random variable maps outcomes to numbers, and its probability distribution describes how likely each value (or range of values) is. There are two flavours, and the difference matters enough to restate precisely before we meet the named distributions.
PMF vs PDF: the one distinction to get right
For a discrete random variable (countable values like ) we use a probability mass function (PMF). The PMF gives the actual probability of each value:
Here each is a probability, so it can never exceed .
For a continuous random variable (any real value, like a height of m) the probability of hitting exactly one value is zero. Instead we use a probability density function (PDF) . Density is "probability per unit length", and we get a probability only by integrating over an interval:
A density is not a probability
A PDF value can be larger than 1. It is a rate, not a probability. For example a Gaussian with peaks at . Only the area under the curve is a probability, and that area is always .
In both cases the total "stuff" sums (discrete) or integrates (continuous) to exactly — that normalization is what makes it a valid distribution.
Expectation and variance
Lesson 01 defined expectation as the probability-weighted average, (or ), and variance as the spread around it. What's new here is the workhorse identity ("mean of the square minus the square of the mean"), which we derive once and reuse for every distribution below:
Where that shortcut comes from
Let . Expand the square and use linearity of expectation (, and is a constant):
The standard deviation has the same units as .
Key identity: — shifting doesn't change variance, scaling does.
Probability distributions
The y-axis is fixed, so parameter changes show their true effect: a wider uniform must get shorter (its area is always 1), and a smaller σ piles the same area into a taller, narrower gaussian peak. The mean (yellow line) is the center of mass; variance is the spread.
Key distributions in ML
Bernoulli: binary outcomes
(think: one coin flip) with success probability . Its PMF is just two numbers:
A neat way to write both at once is for .
Deriving step by step. Sum value probability over the only two outcomes:
Deriving . First . Since and , squaring the values changes nothing here:
Now apply the shortcut :
Note this is largest at (a fair coin is the most uncertain) and zero at or (a certain outcome has no spread).
Where: Binary classification output. A logistic regression outputs a Bernoulli probability.
Categorical: multi-class outcomes
with probabilities , .
Where: Multi-class classification. Softmax output = categorical distribution.
Gaussian (Normal): the bell curve
with and . The normalization constant comes from demanding the total area equal 1, computed via a 2-D polar-coordinate trick. The mean and variance follow from a change of variables and integration by parts.
Why Gaussian is everywhere
The Central Limit Theorem: the mean of many independent random variables converges to a Gaussian, regardless of the original distribution. Features in real datasets are often (approximately) Gaussian — gradients of large neural networks are too.
import numpy as np
rng = np.random.default_rng(42)
# Bernoulli: flip 10000 biased coins, check E[X]=p and Var=p(1-p)
p = 0.7
x = rng.binomial(n=1, p=p, size=10000)
print("Bernoulli E[X]: {:.3f} (true {:.3f})".format(x.mean(), p))
print("Bernoulli Var: {:.3f} (true {:.3f})".format(x.var(), p * (1 - p)))
# Gaussian: check E[X]=mu and Var=sigma^2
mu, sigma = 3.0, 1.5
y = rng.normal(mu, sigma, size=10000)
print("Gaussian E[X]: {:.3f} (true {:.3f})".format(y.mean(), mu))
print("Gaussian Var: {:.3f} (true {:.3f})".format(y.var(), sigma ** 2))
Multivariate Gaussian
The covariance matrix captures both variance (diagonal) and correlations (off-diagonal). It's always symmetric positive semi-definite.
Where: Gaussian Mixture Models, Kalman filters, Gaussian Processes, VAEs.
For the full catalog (Poisson, Exponential, Gamma, Beta, Dirichlet, Laplace, Student-t), with the relationships between them, the conjugate-prior cheat sheet, and a modeling chooser:
Conditional probability and independence
Lesson 01 introduced conditioning as zooming in on the worlds where happened. Here is the same idea as a ratio (the form you'll actually compute with) plus a worked belief update.
Events and are independent if , equivalently .
Worked example. Suppose , , and . What is the probability it is raining given that we see clouds? By Bayes' theorem: Clouds doubled our estimate of rain (from 30% to 60%) because clouds are much more common when it rains than in general.
In ML: Naïve Bayes assumes features are conditionally independent given the class. This strong assumption is often violated but still works well in practice.
The sum and product rules
Combining them gives Bayes' theorem:
Comparing two whole distributions (how far your model sits from the truth ) has its own toolkit (entropy, cross-entropy, and KL divergence), built from scratch in Entropy & KL Divergence.
From distributions to ML: the likelihood
Why does any of this matter for machine learning? Because a model is a distribution with adjustable parameters, and we fit those parameters by asking: "which parameter values make the data we actually observed most probable?" That probability-of-the-data, viewed as a function of the parameters, is the likelihood.
Suppose we observe independent data points drawn from a distribution with parameter . By the product rule, the joint probability factorizes:
Products of many small numbers underflow, so we almost always take the logarithm, turning the product into a sum (the log-likelihood):
Preview: fitting a Bernoulli. Flip a coin times and see heads: which makes that outcome most probable? Intuition says the fraction of heads, , and maximizing confirms it. The next lesson derives that answer step by step and shows the same recipe (write the likelihood, take its log, maximize) is how logistic regression, linear regression, and neural-network classifiers are all trained — their loss functions are negative log-likelihoods in disguise.
Exercise
A die is rolled. What is E[X]?
Need a hint?
E[X] = Σ x · P(X = x) over all outcomes.
Exercise
For X ~ N(μ, σ²), approximately what percentage of values fall within 1 standard deviation of the mean?
Need a hint?
Recall the 68-95-99.7 rule.
Exercise
If A and B are independent events, what is P(A ∩ B)?
Need a hint?
Independence means knowing B tells you nothing about A.
Common mistakes
- Thinking a PDF value is a probability — the PDF can exceed 1 for continuous distributions; only integrals over intervals give probabilities.
- Assuming features are independent when using the independence assumption (naïve Bayes) — this almost never holds exactly.
- Confusing the sample mean converging to the population mean (law of large numbers) with any individual sample being "corrected" by prior outcomes (gambler's fallacy).
Key takeaways
- Distributions summarize uncertainty; ML models output distributions (often Bernoulli or Categorical).
- Expectation = mean; variance = spread; the Gaussian is determined by just two numbers ().
- Conditional probability is the foundation of Bayes' theorem and most probabilistic inference.
- Independence means knowing one variable tells you nothing about the other — rare in real data.
Related concepts
- Maximum Likelihood Estimation: fitting distributions to data
- Bayesian Inference: reasoning about uncertainty with priors
- Probabilistic Models & EM: Gaussian mixtures in practice
- Sampling Methods: how to draw samples from these distributions: inverse transform, rejection, and importance sampling