The Bootstrap: Resampling for Standard Errors & Confidence Intervals
The bootstrap estimates the sampling distribution of almost any statistic by resampling the data you already have — no formula, no normality assumption, no closed form required. When you can compute a statistic, you can bootstrap its standard error and confidence interval. It is the workhorse behind error bars on metrics like AUC, median, and correlation, where analytic standard errors are awkward or unknown.
The plug-in principle
We want to know the sampling variability of a statistic — how much it would wobble if we could redraw the sample from the true distribution over and over. We can't redraw from ; we only have one sample. The bootstrap's move is to substitute the empirical distribution for the unknown (the "plug-in principle"). Sampling from is just drawing from the observed data with replacement. The variability of the statistic across those resamples approximates its true sampling variability.
The algorithm
To bootstrap any statistic :
- Draw a resample of size from the data with replacement — some rows appear multiple times, some not at all.
- Compute the statistic on the resample: .
- Repeat times (– typical).
- The collection is the bootstrap distribution — a stand-in for the sampling distribution of .
The bootstrap standard error is just the standard deviation of that collection:
Why ~63.2% of rows appear in each resample
Drawing times with replacement, the chance a specific row is never picked is . So each resample contains about of the distinct original rows; the rest are duplicates. The left-out 36.8% is exactly the held-out set behind the .632 bootstrap estimator for model evaluation.
Confidence intervals
Three intervals, in increasing order of correctness (and cost):
Percentile interval
Take the empirical and quantiles of the bootstrap distribution directly:
Simple and the most common in practice. It is accurate when the bootstrap distribution is roughly symmetric and unbiased, and it respects the parameter's range (a CI for a correlation stays in ).
Basic (pivotal) interval
Reflects the percentiles around the point estimate, correcting for bias in the bootstrap distribution's location:
Better than percentile when the bootstrap distribution is shifted, but it can produce limits outside the parameter's valid range.
BCa (bias-corrected and accelerated)
The gold standard. It adjusts the percentile endpoints for two things: bias (the bootstrap distribution not centered on ) via a correction , and skew (variance that changes with ) via an acceleration estimated by jackknife. BCa is second-order accurate (its coverage error shrinks like rather than ) and transformation-respecting. Prefer it when the statistic is skewed (ratios, variances, extreme quantiles) and accuracy matters.
| Interval | Corrects bias | Corrects skew | Respects range | Cost |
|---|---|---|---|---|
| Percentile | ✗ | ✗ | ✓ | low |
| Basic | ✓ | ✗ | ✗ | low |
| BCa | ✓ | ✓ | ✓ | higher (jackknife) |
Worked sketch: CI for the median
Analytic standard errors for the median are clumsy (they depend on the unknown density at the median). The bootstrap sidesteps this entirely. Given incomes:
- Resample 200 incomes with replacement; record the median. Repeat times.
- of the 5,000 medians.
- 95% percentile CI = the 2.5th and 97.5th percentiles of those 5,000 medians.
No density estimation, no formula, and the same three lines of code work for the trimmed mean, the Gini coefficient, or AUC.
When the bootstrap fails
The bootstrap is not magic. It breaks when the plug-in distribution is a poor stand-in for in the way the statistic depends on it:
- Extreme order statistics: the maximum or minimum. The sample max can never exceed the largest observed value, so the bootstrap distribution of is degenerate and badly underestimates the upper tail.
- Heavy-tailed data with infinite variance. The bootstrap of the mean is inconsistent when the variance doesn't exist; the resample means don't converge to the right limit.
- Very small . With , is a crude approximation of ; intervals are unreliable regardless of .
- Dependent data (time series, spatial, clustered). Plain resampling destroys the autocorrelation, so the SE is underestimated. Use a block bootstrap (below).
More resamples ≠ more data
Increasing shrinks Monte-Carlo noise in the estimate of the bootstrap distribution, but it cannot add information beyond the original observations. If is too small or is a bad proxy for , no value of fixes it. controls precision of the method; controls what the method can possibly know.
Block bootstrap for time series
Autocorrelated data violates the i.i.d. assumption behind ordinary resampling. The moving block bootstrap preserves short-range dependence: instead of resampling individual points, resample contiguous blocks of length and concatenate them. Within a block the temporal structure is intact; only the joins between blocks are artificial. Choose large enough to capture the autocorrelation length (rules of thumb scale it like ) but small enough to still get many distinct blocks.
Related concepts
- Statistical Inference & Sampling Distributions: the sampling-distribution concept the bootstrap approximates by resampling
- Validation Strategies: cross-validation and the .632 bootstrap are the same resampling idea applied to generalization error
- The Kolmogorov–Smirnov Test: when ties break KS's distribution-free p-value, a permutation/resampling test recovers it
- MCMC Sampling: the other major resampling-driven inference tool, aimed at posteriors rather than sampling distributions