Statistical Inference & Confidence IntervalsOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Statistical Inference & Confidence Intervals

You never measure the whole population — you measure a sample and try to say something about the whole. A model's accuracy on a test set, an A/B test's conversion lift, a poll's support level: each is an estimate from limited data, and each comes with uncertainty. This is the rigor behind the loop's evaluation stage — a single test-set number is meaningless without the error bar this lesson teaches you to compute. Statistical inference is the discipline of turning a sample into a quantified claim about the population. This lesson builds the machinery (sampling distributions, the standard error, and confidence intervals) that the next lesson's hypothesis tests rely on.

Real-world examples: reporting a model's accuracy as "92% ± 1.5%"; deciding whether an A/B test's 2-point lift is real or noise; estimating click-through rate from a day's traffic; error bars on every benchmark you've ever trusted.

Statistic, estimator, and sampling distribution

A statistic is any number computed from a sample — the sample mean xˉ\bar{x}, a proportion p^\hat{p}, a model's measured accuracy. Because the sample is random, the statistic is itself a random variable: draw a different sample and you get a slightly different value.

The distribution of a statistic across all possible samples is its sampling distribution. Its spread is what tells you how much your single estimate might wobble.

xˉ=1n∑i=1nxiis one draw from the sampling distribution of the mean.\bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i \quad\text{is one draw from the sampling distribution of the mean.}

The standard error

The standard error (SE) is the standard deviation of a statistic's sampling distribution — how much the estimate varies from sample to sample. For the sample mean of data with population standard deviation σ\sigma:

  SE(xˉ)=σn  \boxed{\;\text{SE}(\bar{x}) = \frac{\sigma}{\sqrt{n}}\;}

The n\sqrt{n} is the central fact of inference: to halve your uncertainty you need four times the data. Diminishing returns are baked in.

Standard deviation ≠ standard error

σ\sigma describes the spread of individual data points; SE describes the spread of an estimate computed from nn points. SE is always smaller (by n\sqrt{n}) and shrinks with more data, while σ\sigma is a fixed property of the population.

The Central Limit Theorem

Why is the normal distribution everywhere in statistics? The Central Limit Theorem (CLT): the sampling distribution of the mean approaches a normal distribution as nn grows, regardless of the population's shape.

xˉ  →  n→∞    N ⁣(μ,  σ2n)\bar{x} \;\xrightarrow{\;n\to\infty\;}\; \mathcal{N}\!\left(\mu,\; \frac{\sigma^2}{n}\right)

The viz below draws samples from a deliberately skewed population. For n=1n=1 the means just reproduce that skew; raise nn and the histogram of sample means becomes bell-shaped and narrows, tracking the predicted normal — exactly as the CLT promises.

Central Limit Theorem: the sampling distribution of the mean

μ = 0.3300.250.50.751sample mean
pop. SD (σ)0.298
SE = σ/√n0.211
observed spread0.208

Each bar counts how often a sample of size n produced a given mean, over 600 samples drawn from a strongly skewed population. Increase n: the distribution of sample means becomes bell-shaped and narrows, hugging the predicted normal (teal). Its spread is the standard error, SE = σ/√n — the basis of confidence intervals and z-tests.

This is what licenses using normal-based tools (z-scores, confidence intervals, z-tests) even when the underlying data is not normal — as long as nn is reasonably large.

Confidence intervals

A point estimate ("accuracy = 92%") hides its uncertainty. A confidence interval (CI) reports a range of plausible values. Using the CLT, a 95%95\% CI for the mean is:

xˉ  ±  z∗⋅SE,z∗=1.96 for 95%\bar{x} \;\pm\; z^{*}\cdot \text{SE}, \qquad z^{*} = 1.96 \text{ for } 95\%

The multiplier z∗z^* comes from the normal distribution: 95%95\% of its mass lies within 1.961.96 standard deviations of the mean. Common values: z∗=1.645z^* = 1.645 for 90%90\%, 1.961.96 for 95%95\%, 2.5762.576 for 99%99\%.

What '95% confidence' actually means

It is a statement about the procedure, not about one interval. If you repeated the whole sample-and-compute-interval process many times, about 95%95\% of the resulting intervals would contain the true parameter. It is not correct to say "there's a 95% probability the true value is in this interval" — the true value is fixed; the interval is what's random.

Worked example. A model scores 88%88\% accuracy on n=400n = 400 test examples. Treating each prediction as a Bernoulli trial, SE=p^(1−p^)/n=0.88⋅0.12/400≈0.0162\text{SE} = \sqrt{\hat p(1-\hat p)/n} = \sqrt{0.88\cdot0.12/400} \approx 0.0162. The 95%95\% CI is 0.88±1.96(0.0162)=[0.848, 0.912]0.88 \pm 1.96(0.0162) = [0.848,\, 0.912]. So "88%88\%" really means "somewhere around 8585–91%91\%", and a competitor at 90%90\% may not be meaningfully better.

Python
import numpy as np

def accuracy_ci(correct, n, z_star=1.96):
    p_hat = correct / n
    se = np.sqrt(p_hat * (1 - p_hat) / n)   # SE of a proportion
    return p_hat, (p_hat - z_star * se, p_hat + z_star * se)

p_hat, (lo, hi) = accuracy_ci(correct=352, n=400)   # 88% on 400 examples
print(f"accuracy {p_hat:.3f}, 95% CI [{lo:.3f}, {hi:.3f}]")   # [0.848, 0.912]
# A model reporting 0.90 sits inside this interval — not clearly better.

When the statistic has no clean standard-error formula (the median, AUC, a correlation), you can still get a confidence interval by resampling the data itself:

Deep dive
The Bootstrap: Resampling for Standard Errors & Confidence Intervals
Includes runnable notebook

Exercise

A population has standard deviation σ = 20. You take a sample of n = 100. What is the standard error of the sample mean?

010.0020
Need a hint?

SE = σ / √n.

Exercise

The Central Limit Theorem says that as the sample size n grows, the sampling distribution of the mean approaches a normal distribution. What is the crucial condition on the population's own shape?

Need a hint?

Does the population itself have to be normal?

Exercise

A sample mean is 50 with standard error SE = 2. Using z* = 1.96, what is the half-width (the ± margin) of the 95% confidence interval?

05.0010
Need a hint?

margin = z* × SE = 1.96 × 2.

Common mistakes

  1. Confusing SE with SD: error bars on a mean use SE (σ/n\sigma/\sqrt{n}), not the raw σ\sigma.
  2. Reporting a point estimate with no interval: "92% accuracy" on 50 examples is far shakier than on 50,000; the CI makes that visible.
  3. Misreading a CI: the 95%95\% refers to the long-run procedure, not the probability that one fixed interval caught the value.
  4. Assuming normality for tiny nn: the CLT needs a reasonable sample size; for very small nn (and unknown σ\sigma) use the tt-distribution instead.

Key takeaways

  • A statistic computed from a random sample has a sampling distribution; its spread is the standard error, SE=σ/n\text{SE} = \sigma/\sqrt{n}.
  • Uncertainty shrinks only as n\sqrt{n} — 4× the data for half the error.
  • The Central Limit Theorem makes the sampling distribution of the mean approximately normal regardless of the population's shape, which justifies normal-based inference.
  • A confidence interval xˉ±z∗ SE\bar{x} \pm z^{*}\,\text{SE} reports a plausible range; "95% confidence" describes the procedure's long-run coverage, not a single interval.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedGo deeper
Statistical Inference & Confidenc…Probability DistributionsHypothesis Testing & p-va…Maximum Likelihood Estima…Evaluating Clusters & Cho…The Bootstrap: Resampling…