Hypothesis Testing & p-values
Is the new model actually better, or did it just get lucky on the test set? Did the A/B variant really lift conversions, or is the 2-point gap noise? Hypothesis testing is the formal procedure for answering "is this effect real?" It assumes nothing interesting is happening, asks how surprising the data would be under that assumption, and rejects the assumption only when the data are surprising enough. This lesson turns the standard errors and sampling distributions from the previous lesson into decisions.
Real-world examples: A/B tests deciding which feature ships; clinical trials; "is this coefficient significant?" in regression; comparing two models' accuracies; every "p < 0.05" you've seen in a paper.
Null and alternative hypotheses
A test pits two hypotheses against each other:
- Null hypothesis : the "boring" default: no effect, no difference. It's the claim you assume true for the sake of argument, because it pins down a specific distribution to compute with. Always an equality: , or .
- Alternative hypothesis : the effect you actually care about: (two-sided) or (one-sided).
You never 'accept' H₀ — you only fail to reject it
A test can reject or fail to reject it. Failing to reject is not proof that is true — you may simply lack the data to detect the effect. "Absence of evidence is not evidence of absence."
The test statistic and the z-test
A test statistic collapses the data into one number measuring how far the observation sits from , in units of standard error. For a sample mean under a known , the z-statistic is:
Under , this follows the standard normal distribution (thanks to the CLT). A large means the observed mean is many standard errors away from the null value — unlikely if holds. A two-sample z-test compares two groups: .
z-test vs. t-test
Use the z-test when the population is known or is large. When is unknown and estimated from a small sample, the extra uncertainty fattens the tails: use the t-test, whose Student's- distribution approaches the normal as grows (beyond ~30 they're nearly identical). Same logic, slightly wider critical values.
The p-value
The p-value is the probability, assuming is true, of observing a test statistic at least as extreme as the one you got:
Small ⇒ the data would be very surprising under ⇒ evidence against it. The decision rule compares to a pre-chosen significance level (commonly ):
- → reject ("statistically significant")
- → fail to reject
Equivalently, reject when exceeds the critical value . The viz shows this directly: move the observed and , and watch the p-value region (blue) and rejection region (rose) — you reject exactly when the blue spills past the rose boundary.
Hypothesis testing: p-value vs. significance level
The bell is the null distribution of the test statistic. Rose shading is the rejection region (the α-tails); blue shading is the p-value — the probability, if H0 were true, of a statistic at least as extreme as the observed z. Reject H0 exactly when the p-value ≤ α, i.e. when z lands beyond the critical value.
What a p-value is NOT
is not the probability that is true, and is not the probability the effect is real — those require a prior (the Bayesian view). The p-value lives entirely inside the world where holds: means "if there were truly no effect, data this extreme would arise 3% of the time," nothing more.
import numpy as np
from math import erf, sqrt
def normal_cdf(z):
return 0.5 * (1 + erf(z / sqrt(2)))
def z_test(x_bar, mu0, sigma, n):
z = (x_bar - mu0) / (sigma / np.sqrt(n)) # standardize the estimate
p = 2 * (1 - normal_cdf(abs(z))) # two-tailed p-value
return z, p
z, p = z_test(x_bar=105, mu0=100, sigma=20, n=64)
print(f"z = {z:.2f}, p = {p:.4f}") # z = 2.00, p = 0.0455
print("reject H0" if p < 0.05 else "fail to reject H0")
Two ways to be wrong
| true | false | |
|---|---|---|
| Reject | Type I error (rate ) | ✓ correct (power ) |
| Fail to reject | ✓ correct | Type II error (rate ) |
- Type I (false positive): claiming an effect that isn't there; its rate is the you choose.
- Type II (false negative): missing a real effect; its rate falls as sample size or effect size grows.
- Power — the chance of detecting a real effect. More data buys power without raising .
This is exactly the precision/recall-style tension you tune in A/B tests: lowering avoids false alarms but costs power.
For a fuller worked A/B-test trace and the "peeking problem," see the deep dive:
Exercise
In an A/B test comparing conversion rates of variants A and B, what is the correct null hypothesis H₀?
Need a hint?
H₀ is always the 'no effect' claim, stated as an equality.
Exercise
Under H₀ the mean is μ₀ = 100 with σ = 20. You observe a sample mean of 105 from n = 64 samples. What is the z-statistic?
Need a hint?
z = (x̄ − μ₀) / (σ/√n). Here σ/√n = 20/8.
Exercise
A test returns p = 0.03. Which interpretation is correct?
Need a hint?
The p-value is computed entirely inside the world where H₀ is true.
Exercise
A spam filter test uses H₀ = 'this email is not spam.' The filter rejects H₀ (flags the email as spam) for a legitimate email. What type of error is this?
Need a hint?
Rejecting a true null hypothesis.
Common mistakes
- Reading as "probability the null is true." It's the probability of the data given the null, not of the null given the data.
- Treating as proof of no effect. It only means you lacked the evidence; the effect may be real but small or your test underpowered.
- Confusing statistical and practical significance. With huge , a trivial effect can be "significant" yet worthless — always report the effect size and CI too.
- Peeking / multiple comparisons. Checking many times or testing many variants without correction inflates the false-positive rate far above .
Key takeaways
- A test pits (no effect, an equality) against ; you reject or fail to reject — never "accept" it.
- The z-test standardizes an estimate into , which is standard-normal under ; use the t-test when is unknown and is small.
- The p-value is ; reject when . It is not the probability that is true.
- Tests trade Type I (, false positive) against Type II (, false negative); power grows with sample and effect size.
Related concepts
- Statistical Inference & Confidence Intervals: the standard errors and sampling distributions hypothesis tests are built on
- Bayesian Inference: the alternative that gives directly, the quantity p-values are mistaken for
- Continual Learning & Test in Production: A/B testing, power, and the peeking problem in a production ML setting
- The Kolmogorov–Smirnov Test: a distribution-free test that compares two whole distributions via their CDFs, with no binning