Hypothesis Testing: Null, Alternative & p-values
Every A/B test, every "is this model actually better?" question, and every reported "" rests on one piece of machinery: null hypothesis significance testing. The idea is to assume nothing interesting is happening, work out how surprising the observed data would be under that assumption, and reject the assumption only if the data are surprising enough. This page defines the three terms precisely and shows how they connect.
The null hypothesis
The null hypothesis is the "boring" default — the claim that there is no effect, no difference, no relationship. It is the position you assume true for the sake of argument, and the only one you can directly test, because it pins down a specific sampling distribution.
| Setting | Null hypothesis |
|---|---|
| A/B test on conversion rate | the two variants have equal conversion: |
| New model vs. baseline accuracy | the models have equal accuracy: |
| Coin fairness | the coin is fair: |
| Regression coefficient | the feature has no effect: |
is always an equality (or a "no worse than" boundary). That precision is the point: it fixes the distribution of the test statistic, which is what lets you compute probabilities at all.
The alternative hypothesis
The alternative hypothesis (also ) is the claim you actually care about — that an effect does exist. It is the logical complement of over the parameter space, and it determines whether the test is one- or two-sided.
You never 'accept' the alternative — you reject the null
A significance test can only reject or fail to reject it. Failing to reject is not evidence that is true — it may just mean you lacked the data to detect the effect (low power). "Absence of evidence is not evidence of absence."
The test statistic and its null distribution
A test statistic collapses the whole dataset into one number measuring how far the data deviate from , scaled by noise. For comparing two means, the two-sample / statistic is:
Under , follows a known distribution — its null distribution (standard normal for large samples, Student's for small ones). Large means the observed gap is many standard errors away from "no difference," which is unlikely if holds.
The p-value
The p-value is the probability, assuming is true, of observing a test statistic at least as extreme as the one you actually got:
Small ⇒ the data would be very surprising if nothing were going on ⇒ evidence against . The decision rule compares it to a pre-chosen significance level :
- → reject (the result is "statistically significant")
- → fail to reject
What a p-value is NOT
is not the probability that is true, and is not the probability the effect is real — those are Bayesian posterior quantities requiring a prior. is computed entirely inside the world where holds. A p-value of 0.03 means "if there were truly no effect, data this extreme arise 3% of the time," nothing more.
A worked A/B test
Variant A: 200/2000 conversions (10.0%). Variant B: 240/2000 (12.0%). Is B genuinely better, or is a 2-point lift just noise?
vs (two-sided), at .
import numpy as np
from scipy import stats
xA, nA = 200, 2000 # variant A: conversions, total
xB, nB = 240, 2000 # variant B
pA, pB = xA / nA, xB / nB
p_pool = (xA + xB) / (nA + nB) # pooled rate under H0
se = np.sqrt(p_pool * (1 - p_pool) * (1 / nA + 1 / nB))
z = (pB - pA) / se # test statistic
p_value = 2 * (1 - stats.norm.cdf(abs(z))) # two-sided
print(f"rates: A = {pA:.3f} B = {pB:.3f} lift = {pB - pA:+.3f}")
print(f"z statistic = {z:.3f}")
print(f"p-value = {p_value:.4f}")
print("reject H0 (B is better)" if p_value < 0.05 else "fail to reject H0")
# z statistic ≈ 1.961, p-value ≈ 0.0499 -> just barely reject H0
The 2-point lift sits right at the edge of significance — a vivid reminder that "significant" is a threshold artifact, not a hard truth, and that effect size and confidence intervals matter as much as the binary verdict.
The two ways to be wrong
| true | false | |
|---|---|---|
| Reject | Type I error (rate ) | ✓ correct (power ) |
| Fail to reject | ✓ correct | Type II error (rate ) |
- (significance): the false-positive rate you accept up front (typically 0.05). Lowering it makes false alarms rarer but costs power.
- / power: power is the chance of detecting a real effect of a given size. More data raises power without raising .
The peeking problem
The 5% false-positive guarantee assumes you check the p-value once, at a pre-registered sample size. Re-checking every day and stopping on the first is running many tests; the true Type I rate balloons to 20–30%. Fix it by fixing the sample size in advance, or by using sequential / always-valid p-values. (See the A/B-testing lesson.)
Common mistakes
- Reading as "probability the null is true." It is a probability of the data given the null, not of the null given the data.
- Treating as proof of no effect. It only means you didn't gather enough evidence; the effect may be real but small, or your test underpowered.
- Chasing significance by peeking or by testing many variants without a multiple-comparison correction (Bonferroni, Benjamini–Hochberg) — both silently inflate false positives.
- Confusing statistical with practical significance. With a huge , a 0.01% lift can be "significant" yet worthless. Always report the effect size and a confidence interval alongside .
Related concepts
- Probability Distributions: the sampling distributions (normal, ) that supply a test statistic's null distribution
- Bayesian Inference: the Bayesian alternative that gives directly, the quantity a p-value is so often mistaken for
- Continual Learning & Test in Production: A/B testing, power, MDE, and the peeking problem in a production ML setting