Statistical Inference & Confidence Intervals
You never measure the whole population — you measure a sample and try to say something about the whole. A model's accuracy on a test set, an A/B test's conversion lift, a poll's support level: each is an estimate from limited data, and each comes with uncertainty. This is the rigor behind the loop's evaluation stage — a single test-set number is meaningless without the error bar this lesson teaches you to compute. Statistical inference is the discipline of turning a sample into a quantified claim about the population. This lesson builds the machinery (sampling distributions, the standard error, and confidence intervals) that the next lesson's hypothesis tests rely on.
Real-world examples: reporting a model's accuracy as "92% ± 1.5%"; deciding whether an A/B test's 2-point lift is real or noise; estimating click-through rate from a day's traffic; error bars on every benchmark you've ever trusted.
Statistic, estimator, and sampling distribution
A statistic is any number computed from a sample — the sample mean , a proportion , a model's measured accuracy. Because the sample is random, the statistic is itself a random variable: draw a different sample and you get a slightly different value.
The distribution of a statistic across all possible samples is its sampling distribution. Its spread is what tells you how much your single estimate might wobble.
The standard error
The standard error (SE) is the standard deviation of a statistic's sampling distribution — how much the estimate varies from sample to sample. For the sample mean of data with population standard deviation :
The is the central fact of inference: to halve your uncertainty you need four times the data. Diminishing returns are baked in.
Standard deviation ≠ standard error
describes the spread of individual data points; SE describes the spread of an estimate computed from points. SE is always smaller (by ) and shrinks with more data, while is a fixed property of the population.
The Central Limit Theorem
Why is the normal distribution everywhere in statistics? The Central Limit Theorem (CLT): the sampling distribution of the mean approaches a normal distribution as grows, regardless of the population's shape.
The viz below draws samples from a deliberately skewed population. For the means just reproduce that skew; raise and the histogram of sample means becomes bell-shaped and narrows, tracking the predicted normal — exactly as the CLT promises.
Central Limit Theorem: the sampling distribution of the mean
Each bar counts how often a sample of size n produced a given mean, over 600 samples drawn from a strongly skewed population. Increase n: the distribution of sample means becomes bell-shaped and narrows, hugging the predicted normal (teal). Its spread is the standard error, SE = σ/√n — the basis of confidence intervals and z-tests.
This is what licenses using normal-based tools (z-scores, confidence intervals, z-tests) even when the underlying data is not normal — as long as is reasonably large.
Confidence intervals
A point estimate ("accuracy = 92%") hides its uncertainty. A confidence interval (CI) reports a range of plausible values. Using the CLT, a CI for the mean is:
The multiplier comes from the normal distribution: of its mass lies within standard deviations of the mean. Common values: for , for , for .
What '95% confidence' actually means
It is a statement about the procedure, not about one interval. If you repeated the whole sample-and-compute-interval process many times, about of the resulting intervals would contain the true parameter. It is not correct to say "there's a 95% probability the true value is in this interval" — the true value is fixed; the interval is what's random.
Worked example. A model scores accuracy on test examples. Treating each prediction as a Bernoulli trial, . The CI is . So "" really means "somewhere around –", and a competitor at may not be meaningfully better.
import numpy as np
def accuracy_ci(correct, n, z_star=1.96):
p_hat = correct / n
se = np.sqrt(p_hat * (1 - p_hat) / n) # SE of a proportion
return p_hat, (p_hat - z_star * se, p_hat + z_star * se)
p_hat, (lo, hi) = accuracy_ci(correct=352, n=400) # 88% on 400 examples
print(f"accuracy {p_hat:.3f}, 95% CI [{lo:.3f}, {hi:.3f}]") # [0.848, 0.912]
# A model reporting 0.90 sits inside this interval — not clearly better.
When the statistic has no clean standard-error formula (the median, AUC, a correlation), you can still get a confidence interval by resampling the data itself:
Exercise
A population has standard deviation σ = 20. You take a sample of n = 100. What is the standard error of the sample mean?
Need a hint?
SE = σ / √n.
Exercise
The Central Limit Theorem says that as the sample size n grows, the sampling distribution of the mean approaches a normal distribution. What is the crucial condition on the population's own shape?
Need a hint?
Does the population itself have to be normal?
Exercise
A sample mean is 50 with standard error SE = 2. Using z* = 1.96, what is the half-width (the ± margin) of the 95% confidence interval?
Need a hint?
margin = z* × SE = 1.96 × 2.
Common mistakes
- Confusing SE with SD: error bars on a mean use SE (), not the raw .
- Reporting a point estimate with no interval: "92% accuracy" on 50 examples is far shakier than on 50,000; the CI makes that visible.
- Misreading a CI: the refers to the long-run procedure, not the probability that one fixed interval caught the value.
- Assuming normality for tiny : the CLT needs a reasonable sample size; for very small (and unknown ) use the -distribution instead.
Key takeaways
- A statistic computed from a random sample has a sampling distribution; its spread is the standard error, .
- Uncertainty shrinks only as — 4× the data for half the error.
- The Central Limit Theorem makes the sampling distribution of the mean approximately normal regardless of the population's shape, which justifies normal-based inference.
- A confidence interval reports a plausible range; "95% confidence" describes the procedure's long-run coverage, not a single interval.
Related concepts
- Probability Distributions: the normal distribution the CLT and CIs rely on
- Hypothesis Testing: turning these intervals and standard errors into formal decisions
- Maximum Likelihood Estimation: the point-estimation counterpart to interval estimation