Hypothesis Testing & p-valuesOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Hypothesis Testing & p-values

Is the new model actually better, or did it just get lucky on the test set? Did the A/B variant really lift conversions, or is the 2-point gap noise? Hypothesis testing is the formal procedure for answering "is this effect real?" It assumes nothing interesting is happening, asks how surprising the data would be under that assumption, and rejects the assumption only when the data are surprising enough. This lesson turns the standard errors and sampling distributions from the previous lesson into decisions.

Real-world examples: A/B tests deciding which feature ships; clinical trials; "is this coefficient significant?" in regression; comparing two models' accuracies; every "p < 0.05" you've seen in a paper.

Null and alternative hypotheses

A test pits two hypotheses against each other:

  • Null hypothesis H0H_0: the "boring" default: no effect, no difference. It's the claim you assume true for the sake of argument, because it pins down a specific distribution to compute with. Always an equality: μA=μB\mu_A = \mu_B, or p=0.5p = 0.5.
  • Alternative hypothesis H1H_1: the effect you actually care about: μA≠μB\mu_A \ne \mu_B (two-sided) or μB>μA\mu_B > \mu_A (one-sided).

You never 'accept' H₀ — you only fail to reject it

A test can reject H0H_0 or fail to reject it. Failing to reject is not proof that H0H_0 is true — you may simply lack the data to detect the effect. "Absence of evidence is not evidence of absence."

The test statistic and the z-test

A test statistic collapses the data into one number measuring how far the observation sits from H0H_0, in units of standard error. For a sample mean under a known σ\sigma, the z-statistic is:

z=xˉ−μ0SE=xˉ−μ0σ/nz = \frac{\bar{x} - \mu_0}{\text{SE}} = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}}

Under H0H_0, this zz follows the standard normal distribution (thanks to the CLT). A large ∣z∣|z| means the observed mean is many standard errors away from the null value — unlikely if H0H_0 holds. A two-sample z-test compares two groups: z=(xˉB−xˉA)/SEA2+SEB2z = (\bar{x}_B - \bar{x}_A)/\sqrt{\text{SE}_A^2 + \text{SE}_B^2}.

z-test vs. t-test

Use the z-test when the population σ\sigma is known or nn is large. When σ\sigma is unknown and estimated from a small sample, the extra uncertainty fattens the tails: use the t-test, whose Student's-tt distribution approaches the normal as nn grows (beyond ~30 they're nearly identical). Same logic, slightly wider critical values.

The p-value

The p-value is the probability, assuming H0H_0 is true, of observing a test statistic at least as extreme as the one you got:

p=P( ∣Z∣≥∣zobs∣  ∣  H0 )p = P\big(\,|Z| \ge |z_{\text{obs}}| \;\big|\; H_0\,\big)

Small pp ⇒ the data would be very surprising under H0H_0 ⇒ evidence against it. The decision rule compares pp to a pre-chosen significance level α\alpha (commonly 0.050.05):

  • p≤αp \le \alpha → reject H0H_0 ("statistically significant")
  • p>αp > \alpha → fail to reject H0H_0

Equivalently, reject when ∣z∣|z| exceeds the critical value z∗z^*. The viz shows this directly: move the observed zz and α\alpha, and watch the p-value region (blue) and rejection region (rose) — you reject exactly when the blue spills past the rose boundary.

Hypothesis testing: p-value vs. significance level

z = 1.80+z* 1.96-z* -1.96-3-2-10123
p-value0.0719
α0.05
decisionfail to reject

The bell is the null distribution of the test statistic. Rose shading is the rejection region (the α-tails); blue shading is the p-value — the probability, if H0 were true, of a statistic at least as extreme as the observed z. Reject H0 exactly when the p-value ≤ α, i.e. when z lands beyond the critical value.

What a p-value is NOT

pp is not the probability that H0H_0 is true, and 1−p1-p is not the probability the effect is real — those require a prior (the Bayesian view). The p-value lives entirely inside the world where H0H_0 holds: p=0.03p = 0.03 means "if there were truly no effect, data this extreme would arise 3% of the time," nothing more.

Python
import numpy as np
from math import erf, sqrt

def normal_cdf(z):
    return 0.5 * (1 + erf(z / sqrt(2)))

def z_test(x_bar, mu0, sigma, n):
    z = (x_bar - mu0) / (sigma / np.sqrt(n))   # standardize the estimate
    p = 2 * (1 - normal_cdf(abs(z)))           # two-tailed p-value
    return z, p

z, p = z_test(x_bar=105, mu0=100, sigma=20, n=64)
print(f"z = {z:.2f}, p = {p:.4f}")             # z = 2.00, p = 0.0455
print("reject H0" if p < 0.05 else "fail to reject H0")

Two ways to be wrong

H0H_0 trueH0H_0 false
Reject H0H_0Type I error (rate α\alpha)✓ correct (power =1−β=1-\beta)
Fail to reject✓ correctType II error (rate β\beta)
  • Type I (false positive): claiming an effect that isn't there; its rate is the α\alpha you choose.
  • Type II (false negative): missing a real effect; its rate β\beta falls as sample size or effect size grows.
  • Power =1−β= 1-\beta — the chance of detecting a real effect. More data buys power without raising α\alpha.

This is exactly the precision/recall-style tension you tune in A/B tests: lowering α\alpha avoids false alarms but costs power.

For a fuller worked A/B-test trace and the "peeking problem," see the deep dive:

Deep dive
Hypothesis Testing: Null, Alternative & p-values
Includes runnable notebook

Exercise

In an A/B test comparing conversion rates of variants A and B, what is the correct null hypothesis H₀?

Need a hint?

H₀ is always the 'no effect' claim, stated as an equality.

Exercise

Under H₀ the mean is μ₀ = 100 with σ = 20. You observe a sample mean of 105 from n = 64 samples. What is the z-statistic?

02.505
Need a hint?

z = (x̄ − μ₀) / (σ/√n). Here σ/√n = 20/8.

Exercise

A test returns p = 0.03. Which interpretation is correct?

Need a hint?

The p-value is computed entirely inside the world where H₀ is true.

Exercise

A spam filter test uses H₀ = 'this email is not spam.' The filter rejects H₀ (flags the email as spam) for a legitimate email. What type of error is this?

Need a hint?

Rejecting a true null hypothesis.

Common mistakes

  1. Reading pp as "probability the null is true." It's the probability of the data given the null, not of the null given the data.
  2. Treating p>αp > \alpha as proof of no effect. It only means you lacked the evidence; the effect may be real but small or your test underpowered.
  3. Confusing statistical and practical significance. With huge nn, a trivial 0.01%0.01\% effect can be "significant" yet worthless — always report the effect size and CI too.
  4. Peeking / multiple comparisons. Checking many times or testing many variants without correction inflates the false-positive rate far above α\alpha.

Key takeaways

  • A test pits H0H_0 (no effect, an equality) against H1H_1; you reject or fail to reject H0H_0 — never "accept" it.
  • The z-test standardizes an estimate into z=(xˉ−μ0)/SEz = (\bar{x}-\mu_0)/\text{SE}, which is standard-normal under H0H_0; use the t-test when σ\sigma is unknown and nn is small.
  • The p-value is P(data this extreme∣H0)P(\text{data this extreme}\mid H_0); reject when p≤αp \le \alpha. It is not the probability that H0H_0 is true.
  • Tests trade Type I (α\alpha, false positive) against Type II (β\beta, false negative); power =1−β=1-\beta grows with sample and effect size.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedGo deeper
Hypothesis Testing & p-valuesStatistical Inference & C…Bayesian InferenceContinual Learning & Test…Interventions & Potential…The Kolmogorov–Smirnov Te…Hypothesis Testing: Null,…