Probability Distributions: A Field Guide & Where They're Used
Picking the right distribution is half of probabilistic modeling. A distribution encodes your assumption about how a quantity is generated (a count, a duration, a proportion, a category) and that choice flows straight into the likelihood you optimize, the loss you minimize, and the prior you place. This page is a reference catalog: for each distribution, what it models, its parameters and moments, and where it shows up in ML.
The first fork is always the same: is the outcome discrete or continuous?
Discrete distributions
These put probability mass on a countable set of outcomes; they have a PMF that sums to 1.
Bernoulli: a single yes/no
One trial with success probability . , .
- Mean / variance: / (variance is maximal at — maximum uncertainty).
- Where: the output unit of binary classification; every sigmoid + binary-cross-entropy head is a Bernoulli likelihood. A single coin flip, click/no-click, fraud/not-fraud.
Binomial: counting successes in trials
Sum of independent Bernoulli() trials. .
- Mean / variance: / .
- Where: A/B-test conversion counts, number of correct predictions in a batch, sequencing read counts.
Categorical: one draw from classes
The multi-class generalization of Bernoulli: outcome is one of categories with probabilities summing to 1.
- Where: the softmax output of every multi-class classifier and every language model — predicting the next token is a Categorical over the vocabulary. Multinomial is its -trial count version (a bag-of-words vector is multinomial).
Poisson: counts of rare events in a fixed window
Number of events when they occur independently at a constant average rate . .
- Mean / variance: both equal (the signature property).
- Where: Poisson regression for count targets (arrivals per hour, defects per unit, clicks per impression). When the data's variance exceeds its mean (overdispersion), reach for the Negative Binomial instead.
Geometric & Negative Binomial: waiting for successes
Geometric: number of trials until the first success. Negative Binomial: trials until the -th success (and the overdispersed count model above).
- Where: churn / time-to-event in discrete steps; count data with variance > mean.
Continuous distributions
These have a PDF that integrates to 1; is a density, not a probability (it can exceed 1).
Uniform: maximal ignorance on an interval
Constant density on . Where: non-informative priors, the seed of inverse-transform sampling (every sampler starts from Uniform(0,1)), random initialization ranges, dropout masks' threshold.
Gaussian (Normal): the default for continuous quantities
The bell curve . Justified by the Central Limit Theorem: sums of many small independent effects are approximately Gaussian.
- Mean / variance: / .
- Where: everywhere — regression noise (MSE loss is a Gaussian likelihood), weight initialization, VAE latent priors, Gaussian Processes, Kalman filters, diffusion-model noise. The Multivariate Gaussian extends it to correlated vectors via a covariance matrix .
MSE is a Gaussian assumption in disguise
Minimizing squared error is exactly maximum-likelihood under "targets = prediction + Gaussian noise of constant variance." If your residuals are heavy-tailed or skewed, that assumption is wrong, which is why a Laplace likelihood (→ MAE loss) or a Student-t likelihood (robust regression) can fit better. The loss function you pick is a distributional assumption.
Exponential & Gamma: durations and positive quantities
Exponential models the waiting time between Poisson events (rate , memoryless). Gamma is the time until the -th event and, more generally, a flexible distribution over positive reals.
- Where: survival / time-to-event modeling, the conjugate prior for the rate of a Poisson or precision of a Gaussian (Gamma), inter-arrival times.
Beta: a distribution over a probability
Defined on with shape parameters ; its mean is . Because it lives on the unit interval, it models an uncertain probability.
- Where: the conjugate prior for Bernoulli/Binomial — Beta prior + Binomial data → Beta posterior (the backbone of Bayesian A/B testing and Thompson sampling for bandits).
Dirichlet: a distribution over distributions
The multivariate Beta: samples are probability vectors on the simplex (they sum to 1).
- Where: the conjugate prior for the Categorical/Multinomial — topic models (LDA), Bayesian mixture weights, smoothing categorical estimates.
Laplace & Student-t: heavy tails and robustness
Laplace (double-exponential) is peaked with heavier tails than Gaussian; Student-t has tunable tail weight via its degrees of freedom (→ Gaussian as df → ∞).
- Where: Laplace prior ↔ L1 / Lasso sparsity; Laplace likelihood ↔ MAE loss; Student-t for robust regression and heavy-tailed financial returns where Gaussian under-models extremes.
Chi-squared, F: the sampling distributions behind tests
Sums of squared Gaussians (χ²) and ratios of them (F) — the reference distributions for variance and ANOVA tests rather than data-generating models.
How they connect
These are not isolated; most are special or limiting cases of one another:
- Bernoulli is Binomial with ; Categorical is the -outcome Bernoulli; Multinomial is Categorical draws.
- Binomial → Poisson as , with fixed (rare-event limit).
- Sum of many anything → Gaussian (Central Limit Theorem).
- Exponential is Gamma with shape 1; Gamma → Gaussian for large shape.
- Beta generalizes to Dirichlet; Gaussian generalizes to Multivariate Gaussian.
The conjugate-prior cheat sheet
Conjugacy means the posterior stays in the same family as the prior — closed-form Bayesian updates with no MCMC. The pairs worth memorizing:
| Likelihood | Conjugate prior | Posterior | Used for |
|---|---|---|---|
| Bernoulli / Binomial | Beta | Beta | A/B tests, bandits |
| Categorical / Multinomial | Dirichlet | Dirichlet | topic models, smoothing |
| Poisson | Gamma | Gamma | count-rate estimation |
| Gaussian (known var), mean | Gaussian | Gaussian | Kalman filters, online means |
| Gaussian (unknown var) | Inverse-Gamma | Inverse-Gamma | variance estimation |
A modeling chooser
Start from what the quantity is, not from what's convenient:
- Binary outcome → Bernoulli (sigmoid head).
- One of K classes → Categorical (softmax head).
- Count in a fixed window → Poisson (or Negative Binomial if overdispersed).
- Continuous, symmetric, light tails → Gaussian (MSE).
- Continuous, heavy tails / outliers → Student-t or Laplace (robust loss).
- Positive-only (durations, magnitudes) → Exponential / Gamma / Log-Normal.
- A proportion in [0,1] → Beta.
- A probability vector on the simplex → Dirichlet.
Match the support to the data
The most common modeling bug is a distribution whose support doesn't match the data's range: a Gaussian likelihood on strictly-positive durations puts mass on impossible negative values; a Gaussian on a bounded proportion ignores the ceiling. Check the support first — it rules out most candidates immediately.
Related concepts
- Probability Distributions: the lesson deriving PMFs/PDFs, expectation, and variance
- Thinking in Probabilities: the foundations these distributions are built on
- Sampling Methods: how to draw samples from any of these distributions
- Beta–Bernoulli Conjugate Posterior: the Beta/Bernoulli conjugate update worked end to end
- The Gaussian Integral: why the normal distribution's normalizing constant is what it is