Validation StrategiesOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Validation Strategies

A model that scores 98% on its training data might generalize at 60%, or the reverse. The only way to measure true generalization is to test on data the model never saw during training. But that simple principle hides a minefield of ways to accidentally peek at test data.

The fundamental goal

We want to estimate expected loss on unseen data from a distribution D\mathcal{D}:

Lgen=E(x,y)∼D[ℓ(f(x),y)]L_{\text{gen}} = \mathbb{E}_{(x, y) \sim \mathcal{D}}[\ell(f(x), y)]

The training loss LtrainL_{\text{train}} is a biased (optimistic) estimate — the model was fit to minimize it. The question is: how do we estimate LgenL_{\text{gen}} from our finite dataset without holding back too much data from training?

Simple train/test split

The simplest strategy: reserve 20% as a held-out test set, train on 80%.

Stratified split: for classification, preserve the class distribution in both splits. Without stratification, random chance might put all rare-class examples in one set.

When it's sufficient: large datasets (>10k samples). With enough data, the 20% test set is large enough that the estimate has low variance.

When it fails: small datasets. A single 80/20 split on 100 samples leaves 20 test examples — the estimated test accuracy has high variance (~3-5% standard deviation).

Never touch the test set during development

The test set must only be used once — after all model development is done. If you tune hyperparameters based on test performance, you are implicitly fitting the test set and the estimate is no longer unbiased. Use a validation set for development; the test set is for the final evaluation only.

k-Fold Cross-Validation

Partition the dataset into kk equal folds. For each fold ii, train on the other k−1k-1 folds and evaluate on fold ii. Repeat for all folds and average the scores.

L^CV=1k∑i=1kLvali\hat{L}_{\text{CV}} = \frac{1}{k} \sum_{i=1}^k L_{\text{val}_i}

Choosing k:

kBiasVarianceCost
2High (trains on 50%)LowCheap
5MediumMediumModerate
10LowLow-medium10× training cost
n (LOO)Very lowHighn× training cost

k=5 or k=10 is the standard recommendation: low bias (trains on 80-90% of data) with manageable compute.

Stratified k-fold preserves class ratios in each fold — always use for classification.

Python
from sklearn.model_selection import StratifiedKFold, cross_val_score

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=skf, scoring='f1')
print(f"CV F1: {scores.mean():.3f} ± {scores.std():.3f}")

Leave-One-Out (LOO)

LOO is k-fold with k=nk = n: train on n−1n-1 samples and test on the one held-out sample, repeated nn times.

Advantages: nearly unbiased estimate (trains on almost all the data).

Disadvantages: nn separate model fits (expensive); the nn validation estimates are highly correlated (each fold differs by just one sample), giving the LOO estimate high variance.

Use LOO only when your dataset is very small (n<50n < 50) and compute is cheap.

Bootstrapping

Bootstrap sampling: draw nn samples with replacement from the nn-sample dataset. Each bootstrap sample is used for training; the samples not drawn (out-of-bag, OOB) form the validation set.

OOB fraction: the probability that a specific sample is never chosen in nn draws is (1−1/n)n→e−1≈0.368(1-1/n)^n \to e^{-1} \approx 0.368 as n→∞n \to \infty. About 36.8% of the original data is OOB per bootstrap sample — equivalent to a roughly 63/37 train/test split.

The .632 estimator corrects the bootstrap's slight optimism:

L^0.632=0.368⋅Ltrain+0.632⋅LOOB\hat{L}_{0.632} = 0.368 \cdot L_{\text{train}} + 0.632 \cdot L_{\text{OOB}}

Random Forest OOB error: Random Forest builds each tree on a bootstrap sample, so it automatically has OOB samples for each tree — a free, unbiased generalization estimate with no extra training.

Time-series cross-validation

Standard k-fold randomly shuffles the data — catastrophic for time series where future data cannot inform past predictions.

Walk-forward validation: train on [1,t][1, t], validate on [t+1,t+h][t+1, t+h], then expand the window and repeat.

Code
Train [────────]  Val [──]
Train [──────────]  Val [──]
Train [────────────]  Val [──]

Expanding window (above): always train on all past data. Rolling window: use a fixed-length training window, useful when concept drift makes old data less relevant.

Never shuffle time-series data

Random k-fold on time-series data produces optimistic estimates because models see future information during training. A model predicting stock prices using tomorrow's prices as a feature would look great on shuffled CV but fail catastrophically in production.

All of them, on the same forty rows

Every scheme above is a rule for colouring one table: which rows this fit trains on, which it is scored on, which it ignores. Put them on the same 40 rows and the differences stop being descriptions.

Every split, on the same forty rows

row 1row 40 — time →fold 1fold 2fold 3fold 4fold 5trainvalidateunused
model fits5
trains on80.0%
validates on20.0%
rows validated using their own future39 of 40
folds with no positive1 of 5

Columns are the 40 rows of one dataset, always in time order; each band is one fold of the chosen scheme. Teal is validated, grey is trained on, dark is unused by that fold. Yellow ticks above mark the 5 rows of the positive class. Every count below is read off the layout you are looking at.

The columns are always time order, never fold order, which is what makes shuffled k-fold render as speckle and walk-forward render as a staircase. The counts underneath are read off the layout on screen, not quoted.

The bias/variance table, as pictures. Move k from 2 to 10 on k-fold: the grey area per row grows from 50% to 90% (less bias — each fit sees more data) and the teal area shrinks from 20 rows to 4 (more variance — each score is an average over fewer examples). Switch to leave-one-out and the grid becomes 40 rows of one teal cell each: 97.5% training data, and 40 fits to get it. The panel also makes LOO's variance problem concrete rather than abstract — 35 of its 40 folds contain no positive, because there are only 5 in the dataset. Each fold's score is a single 00 or 11, so precision and recall are undefined almost everywhere and only the average means anything. That is what "nearly unbiased, high variance" looks like on a classification task.

Why stratification is not optional. Five of the forty rows are positive, marked in yellow. At k = 5, stratified k-fold puts exactly one in every fold, always. Plain k-fold does not — the shuffle on screen leaves one fold empty, and across 4000 random shuffles 30.9% of folds contained no positive on average, with 95.3% of shuffles producing at least one such fold. A fold with no positive has undefined precision and recall, so the CV mean becomes an average over a metric that did not exist for some of its terms. This is the case scikit-learn warns about, and it is the common outcome rather than the unlucky one.

Where the leakage counter bites. A validation row is contaminated if the same fit trained on any row later in time. Shuffled 5-fold contaminates 39 of 40 rows. The obvious repair (press "keep time order" so folds are contiguous blocks) only gets to 32 of 40, and that is not bad luck:

#{(v,t):t>v,  different folds}  =  (402)−k(40/k2)\#\{(v,t) : t > v,\; \text{different folds}\} \;=\; \binom{40}{2} - k\binom{40/k}{2}

which is 640640 at k=5k = 5 whichever order the rows are in. Ordering the folds does not remove leakage; it concentrates it into the early folds and leaves the last one clean. Only walk-forward reaches 00 of 4040, and that is the entire reason it exists.

Walk-forward pays for it, and the same panel shows the bill: it fits on 43.8%43.8\% of the data on average instead of 80%80\%, discards everything after each validation window, and (because its horizon here is 5 rows against 5 positives in the whole dataset) leaves 3 of its 6 windows with no positive in them. Rare events on short horizons are the standard failure of walk-forward, and the fix is a longer horizon or pooling the predictions across splits before scoring, not a return to shuffling.

Bootstrap's 36.8% is a limit, not a number. The out-of-bag share is (1−1/n)n(1 - 1/n)^n, which is 36.32%36.32\% at n=40n = 40 and only approaches 1/e=36.79%1/e = 36.79\% as n→∞n \to \infty — verified over 20,000 simulated resamples (0.36320.3632 at n=40n=40, 0.36680.3668 at n=200n=200, 0.36760.3676 at n=1000n=1000). Both numbers are on the panel, and the honest lesson is in the gap between them: with only six resamples drawn the measured share reads 37.1%37.1\%, which is roughly one standard error from either candidate. Six draws cannot resolve a half-point difference, and the readout is not asking you to pretend otherwise.

Data leakage

Data leakage is any process that allows information from the validation or test set to influence the training pipeline. It produces optimistic estimates that fail in production.

Three common types:

1. Train-test leakage (preprocessing leakage):

Python
# WRONG — scaler sees test statistics during fit
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_all)   # X_all includes test!
X_train, X_test = train_test_split(X_scaled)

# CORRECT — scaler only sees training data
X_train, X_test = train_test_split(X_all)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test  = scaler.transform(X_test)       # transform only, not fit

2. Temporal leakage: a feature like "days since last fraud in account" computed over the entire dataset uses future fraud events to describe past transactions. Must compute using only data up to each timestamp.

3. Target leakage: a feature that encodes the label. Example: including "treatment prescribed" as a feature when predicting "disease diagnosed" — the prescription was made because of the diagnosis.

Use sklearn Pipeline to prevent leakage

Wrapping preprocessing in a Pipeline ensures every transformer is fit only on the training fold during CV — no leakage possible.

Python
from sklearn.pipeline import Pipeline
pipe = Pipeline([('scaler', StandardScaler()), ('clf', LogisticRegression())])
cross_val_score(pipe, X, y, cv=5)  # scaler fit per fold automatically

Exercise

You run 5-fold CV on a small dataset of 100 samples. Compared to leave-one-out CV (LOO), what is the tradeoff of 5-fold?

Need a hint?

LOO trains on n-1 samples; 5-fold trains on 80. Which has higher bias and which has higher variance?

Exercise

Which of these pipelines contains data leakage?

Need a hint?

Leakage occurs when information from the validation/test set influences the training process.

Exercise

In bootstrap sampling (sampling n points with replacement from n), approximately what fraction of original samples will NOT be included in a given bootstrap sample (the out-of-bag fraction)?

Need a hint?

The probability that a specific sample is not chosen in any of n draws is (1 - 1/n)^n. What does this converge to?

Common mistakes

  1. Tuning hyperparameters on the test set: even reading the test set error once to decide something is leakage. Use a separate validation set or CV.
  2. Not stratifying splits on imbalanced data: all rare-class examples in one fold causes wildly unstable CV estimates.
  3. Applying k-fold to time series: shuffled CV inflates performance on temporal data.
  4. Feature selection before splitting: selecting features based on correlation with labels computed on all data (including test) is leakage. Feature selection must happen inside each CV fold.
  5. Reporting mean CV score without standard deviation: the mean alone hides variance between folds; always report mean ± std.

Key takeaways

  • Training loss is always an optimistic estimate of generalization; validation on held-out data is essential.
  • k-fold CV with k=5 or k=10 balances bias (trains on most data) and variance (averages over multiple splits).
  • LOO is nearly unbiased but has high variance and cost; use for very small datasets.
  • Bootstrap OOB fraction is ≈1/e ≈ 36.8%; the .632 estimator corrects for bootstrapping's slight optimism.
  • Time-series data requires walk-forward validation — never shuffle temporal data.
  • Data leakage (preprocessing, temporal, or target) makes validation estimates optimistic and untrustworthy in production.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Validation StrategiesLinear & Logistic Regress…KNN & Decision TreesNeural NetworksHyperparameter Optimizati…Classification & Regressi…Training Techniques & Hyp…Learning Theory & General…Rank Ordering & Model Com…Feature EngineeringROC Curve and AUC: Deriva…Is That Difference Real? …The Bootstrap: Resampling…