Hypothesis Testing: Null, Alternative & p-valuesOpen in Colab

Hypothesis Testing: Null, Alternative & p-values

Every A/B test, every "is this model actually better?" question, and every reported "p<0.05p < 0.05" rests on one piece of machinery: null hypothesis significance testing. The idea is to assume nothing interesting is happening, work out how surprising the observed data would be under that assumption, and reject the assumption only if the data are surprising enough. This page defines the three terms precisely and shows how they connect.

The null hypothesis H0H_0

The null hypothesis is the "boring" default — the claim that there is no effect, no difference, no relationship. It is the position you assume true for the sake of argument, and the only one you can directly test, because it pins down a specific sampling distribution.

SettingNull hypothesis H0H_0
A/B test on conversion ratethe two variants have equal conversion: pA=pBp_A = p_B
New model vs. baseline accuracythe models have equal accuracy: μnew=μbase\mu_{\text{new}} = \mu_{\text{base}}
Coin fairnessthe coin is fair: p=0.5p = 0.5
Regression coefficientthe feature has no effect: βj=0\beta_j = 0

H0H_0 is always an equality (or a "no worse than" boundary). That precision is the point: it fixes the distribution of the test statistic, which is what lets you compute probabilities at all.

The alternative hypothesis H1H_1

The alternative hypothesis (also HAH_A) is the claim you actually care about — that an effect does exist. It is the logical complement of H0H_0 over the parameter space, and it determines whether the test is one- or two-sided.

two-sided: H1:pA≠pBone-sided: H1:pB>pA\text{two-sided: } H_1: p_A \ne p_B \qquad \text{one-sided: } H_1: p_B > p_A

You never 'accept' the alternative — you reject the null

A significance test can only reject H0H_0 or fail to reject it. Failing to reject is not evidence that H0H_0 is true — it may just mean you lacked the data to detect the effect (low power). "Absence of evidence is not evidence of absence."

The test statistic and its null distribution

A test statistic TT collapses the whole dataset into one number measuring how far the data deviate from H0H_0, scaled by noise. For comparing two means, the two-sample zz/tt statistic is:

T=θ^−θ0SE^=xˉB−xˉA SE^A2+SE^B2 T = \frac{\hat{\theta} - \theta_0}{\widehat{\text{SE}}} = \frac{\bar{x}_B - \bar{x}_A}{\sqrt{\,\widehat{\text{SE}}_A^2 + \widehat{\text{SE}}_B^2\,}}

Under H0H_0, TT follows a known distribution — its null distribution (standard normal for large samples, Student's tt for small ones). Large ∣T∣|T| means the observed gap is many standard errors away from "no difference," which is unlikely if H0H_0 holds.

The p-value

The p-value is the probability, assuming H0H_0 is true, of observing a test statistic at least as extreme as the one you actually got:

  p=P( ∣T∣≥∣tobs∣  ∣  H0 )  \boxed{\; p = P\big(\,|T| \ge |t_{\text{obs}}| \;\big|\; H_0 \,\big) \;}

Small pp ⇒ the data would be very surprising if nothing were going on ⇒ evidence against H0H_0. The decision rule compares it to a pre-chosen significance level α\alpha:

  • p≤αp \le \alpha → reject H0H_0 (the result is "statistically significant")
  • p>αp > \alpha → fail to reject H0H_0

What a p-value is NOT

pp is not the probability that H0H_0 is true, and 1−p1 - p is not the probability the effect is real — those are Bayesian posterior quantities requiring a prior. pp is computed entirely inside the world where H0H_0 holds. A p-value of 0.03 means "if there were truly no effect, data this extreme arise 3% of the time," nothing more.

A worked A/B test

Variant A: 200/2000 conversions (10.0%). Variant B: 240/2000 (12.0%). Is B genuinely better, or is a 2-point lift just noise?

H0:pA=pBH_0: p_A = p_B  vs  H1:pA≠pBH_1: p_A \ne p_B (two-sided), at α=0.05\alpha = 0.05.

Python
import numpy as np
from scipy import stats

xA, nA = 200, 2000     # variant A: conversions, total
xB, nB = 240, 2000     # variant B

pA, pB = xA / nA, xB / nB
p_pool = (xA + xB) / (nA + nB)                       # pooled rate under H0
se = np.sqrt(p_pool * (1 - p_pool) * (1 / nA + 1 / nB))
z = (pB - pA) / se                                   # test statistic
p_value = 2 * (1 - stats.norm.cdf(abs(z)))           # two-sided

print(f"rates:    A = {pA:.3f}   B = {pB:.3f}   lift = {pB - pA:+.3f}")
print(f"z statistic = {z:.3f}")
print(f"p-value     = {p_value:.4f}")
print("reject H0 (B is better)" if p_value < 0.05 else "fail to reject H0")
# z statistic ≈ 1.961,  p-value ≈ 0.0499  ->  just barely reject H0

The 2-point lift sits right at the edge of significance — a vivid reminder that "significant" is a threshold artifact, not a hard truth, and that effect size and confidence intervals matter as much as the binary verdict.

The two ways to be wrong

H0H_0 trueH0H_0 false
Reject H0H_0Type I error (rate α\alpha)✓ correct (power =1−β= 1-\beta)
Fail to reject✓ correctType II error (rate β\beta)
  • α\alpha (significance): the false-positive rate you accept up front (typically 0.05). Lowering it makes false alarms rarer but costs power.
  • β\beta / power: power 1−β1-\beta is the chance of detecting a real effect of a given size. More data raises power without raising α\alpha.

The peeking problem

The 5% false-positive guarantee assumes you check the p-value once, at a pre-registered sample size. Re-checking every day and stopping on the first p<0.05p < 0.05 is running many tests; the true Type I rate balloons to 20–30%. Fix it by fixing the sample size in advance, or by using sequential / always-valid p-values. (See the A/B-testing lesson.)

Common mistakes

  1. Reading pp as "probability the null is true." It is a probability of the data given the null, not of the null given the data.
  2. Treating p>αp > \alpha as proof of no effect. It only means you didn't gather enough evidence; the effect may be real but small, or your test underpowered.
  3. Chasing significance by peeking or by testing many variants without a multiple-comparison correction (Bonferroni, Benjamini–Hochberg) — both silently inflate false positives.
  4. Confusing statistical with practical significance. With a huge NN, a 0.01% lift can be "significant" yet worthless. Always report the effect size and a confidence interval alongside pp.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Hypothesis Testing: Null, Alterna…Is That Difference Real? …Random WalkThe Augmented Dickey-Full…The Kolmogorov–Smirnov Te…Hypothesis Testing & p-va…Continual Learning & Test…Probability DistributionsBayesian Inference