Probability Distributions: A Field Guide & Where They're UsedOpen in Colab

Probability Distributions: A Field Guide & Where They're Used

Picking the right distribution is half of probabilistic modeling. A distribution encodes your assumption about how a quantity is generated (a count, a duration, a proportion, a category) and that choice flows straight into the likelihood you optimize, the loss you minimize, and the prior you place. This page is a reference catalog: for each distribution, what it models, its parameters and moments, and where it shows up in ML.

The first fork is always the same: is the outcome discrete or continuous?

Discrete distributions

These put probability mass on a countable set of outcomes; they have a PMF P(X=k)P(X=k) that sums to 1.

Bernoulli: a single yes/no

One trial with success probability pp. P(X=1)=pP(X=1)=p, P(X=0)=1−pP(X=0)=1-p.

  • Mean / variance: pp / p(1−p)p(1-p) (variance is maximal at p=0.5p=0.5 — maximum uncertainty).
  • Where: the output unit of binary classification; every sigmoid + binary-cross-entropy head is a Bernoulli likelihood. A single coin flip, click/no-click, fraud/not-fraud.

Binomial: counting successes in nn trials

Sum of nn independent Bernoulli(pp) trials. P(X=k)=(nk)pk(1−p)n−kP(X=k)=\binom{n}{k}p^k(1-p)^{n-k}.

  • Mean / variance: npnp / np(1−p)np(1-p).
  • Where: A/B-test conversion counts, number of correct predictions in a batch, sequencing read counts.

Categorical: one draw from KK classes

The multi-class generalization of Bernoulli: outcome is one of KK categories with probabilities π\boldsymbol{\pi} summing to 1.

  • Where: the softmax output of every multi-class classifier and every language model — predicting the next token is a Categorical over the vocabulary. Multinomial is its nn-trial count version (a bag-of-words vector is multinomial).

Poisson: counts of rare events in a fixed window

Number of events when they occur independently at a constant average rate λ\lambda. P(X=k)=λke−λk!P(X=k)=\frac{\lambda^k e^{-\lambda}}{k!}.

  • Mean / variance: both equal λ\lambda (the signature property).
  • Where: Poisson regression for count targets (arrivals per hour, defects per unit, clicks per impression). When the data's variance exceeds its mean (overdispersion), reach for the Negative Binomial instead.

Geometric & Negative Binomial: waiting for successes

Geometric: number of trials until the first success. Negative Binomial: trials until the rr-th success (and the overdispersed count model above).

  • Where: churn / time-to-event in discrete steps; count data with variance > mean.

Continuous distributions

These have a PDF p(x)p(x) that integrates to 1; p(x)p(x) is a density, not a probability (it can exceed 1).

Uniform: maximal ignorance on an interval

Constant density on [a,b][a,b]. Where: non-informative priors, the seed of inverse-transform sampling (every sampler starts from Uniform(0,1)), random initialization ranges, dropout masks' threshold.

Gaussian (Normal): the default for continuous quantities

The bell curve N(μ,σ2)\mathcal{N}(\mu,\sigma^2). Justified by the Central Limit Theorem: sums of many small independent effects are approximately Gaussian.

  • Mean / variance: μ\mu / σ2\sigma^2.
  • Where: everywhere — regression noise (MSE loss is a Gaussian likelihood), weight initialization, VAE latent priors, Gaussian Processes, Kalman filters, diffusion-model noise. The Multivariate Gaussian extends it to correlated vectors via a covariance matrix Σ\boldsymbol{\Sigma}.

MSE is a Gaussian assumption in disguise

Minimizing squared error is exactly maximum-likelihood under "targets = prediction + Gaussian noise of constant variance." If your residuals are heavy-tailed or skewed, that assumption is wrong, which is why a Laplace likelihood (→ MAE loss) or a Student-t likelihood (robust regression) can fit better. The loss function you pick is a distributional assumption.

Exponential & Gamma: durations and positive quantities

Exponential models the waiting time between Poisson events (rate λ\lambda, memoryless). Gamma is the time until the kk-th event and, more generally, a flexible distribution over positive reals.

  • Where: survival / time-to-event modeling, the conjugate prior for the rate of a Poisson or precision of a Gaussian (Gamma), inter-arrival times.

Beta: a distribution over a probability

Defined on [0,1][0,1] with shape parameters α,β\alpha,\beta; its mean is α/(α+β)\alpha/(\alpha+\beta). Because it lives on the unit interval, it models an uncertain probability.

  • Where: the conjugate prior for Bernoulli/Binomial — Beta prior + Binomial data → Beta posterior (the backbone of Bayesian A/B testing and Thompson sampling for bandits).

Dirichlet: a distribution over distributions

The multivariate Beta: samples are probability vectors on the simplex (they sum to 1).

  • Where: the conjugate prior for the Categorical/Multinomial — topic models (LDA), Bayesian mixture weights, smoothing categorical estimates.

Laplace & Student-t: heavy tails and robustness

Laplace (double-exponential) is peaked with heavier tails than Gaussian; Student-t has tunable tail weight via its degrees of freedom (→ Gaussian as df → ∞).

  • Where: Laplace prior ↔ L1 / Lasso sparsity; Laplace likelihood ↔ MAE loss; Student-t for robust regression and heavy-tailed financial returns where Gaussian under-models extremes.

Chi-squared, F: the sampling distributions behind tests

Sums of squared Gaussians (χ²) and ratios of them (F) — the reference distributions for variance and ANOVA tests rather than data-generating models.

How they connect

These are not isolated; most are special or limiting cases of one another:

  • Bernoulli is Binomial with n=1n=1; Categorical is the KK-outcome Bernoulli; Multinomial is nn Categorical draws.
  • Binomial → Poisson as n→∞n\to\infty, p→0p\to 0 with np=λnp=\lambda fixed (rare-event limit).
  • Sum of many anything → Gaussian (Central Limit Theorem).
  • Exponential is Gamma with shape 1; Gamma → Gaussian for large shape.
  • Beta generalizes to Dirichlet; Gaussian generalizes to Multivariate Gaussian.

The conjugate-prior cheat sheet

Conjugacy means the posterior stays in the same family as the prior — closed-form Bayesian updates with no MCMC. The pairs worth memorizing:

LikelihoodConjugate priorPosteriorUsed for
Bernoulli / BinomialBetaBetaA/B tests, bandits
Categorical / MultinomialDirichletDirichlettopic models, smoothing
PoissonGammaGammacount-rate estimation
Gaussian (known var), meanGaussianGaussianKalman filters, online means
Gaussian (unknown var)Inverse-GammaInverse-Gammavariance estimation

A modeling chooser

Start from what the quantity is, not from what's convenient:

  • Binary outcome → Bernoulli (sigmoid head).
  • One of K classes → Categorical (softmax head).
  • Count in a fixed window → Poisson (or Negative Binomial if overdispersed).
  • Continuous, symmetric, light tails → Gaussian (MSE).
  • Continuous, heavy tails / outliers → Student-t or Laplace (robust loss).
  • Positive-only (durations, magnitudes) → Exponential / Gamma / Log-Normal.
  • A proportion in [0,1] → Beta.
  • A probability vector on the simplex → Dirichlet.

Match the support to the data

The most common modeling bug is a distribution whose support doesn't match the data's range: a Gaussian likelihood on strictly-positive durations puts mass on impossible negative values; a Gaussian on a bounded proportion ignores the [0,1][0,1] ceiling. Check the support first — it rules out most candidates immediately.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Probability Distributions: A Fiel…Information Theory for ML…Sampling Methods: Inverse…Beta-Bernoulli Conjugate …The Gaussian Integral (No…Probability DistributionsThinking in Probabilities