Bagging & Random Forests
A single decision tree overfits easily — it's high-variance. Bagging (Bootstrap Aggregating) trains many trees on random subsets of data and averages their predictions, dramatically reducing variance. In project-loop terms, an ensemble is a hypothesis-space move: instead of one model, your function family becomes averages of many — strictly more expressive, and much lower variance.
Real-world examples: Credit scoring, customer churn prediction, medical diagnosis, any tabular data task.
Bagging (Bootstrap Aggregating)
- Create bootstrap samples (sample with replacement, same size as original)
- Train a decision tree on each sample
- Classification: majority vote. Regression: average predictions.
Each tree sees a different "view" of the data. Some points appear multiple times, others not at all (~37% are left out per tree). Averaging many diverse trees reduces variance without increasing bias.
Watch how bootstrap resampling creates diverse trees, and how majority vote smooths the combined boundary:
Bagging: bootstrap resampling & majority-vote boundary
Left: each tree sees a bootstrap resample (faint = not sampled, bold = included). Right: the majority-vote boundary smooths out as more trees are added — high-variance individual stumps average into a stable ensemble.
Why bagging works
Different bootstrap samples produce different trees, and averaging diverse estimates cancels their individual noise — like polling: one poll is noisy, but averaging 1000 polls gives a reliable estimate. If the trees were perfectly uncorrelated, the variance of the average would drop by a full factor of ; the next section shows how close bagging actually gets.
How much does variance actually drop?
The "factor of " only holds if the trees are perfectly uncorrelated, but bagged trees aren't, since they share most of their data. For identically-distributed trees each with variance and pairwise correlation , the variance of their average is:
Two terms with very different behavior:
- The second term vanishes as — adding more trees kills it.
- The first term is a floor that more trees cannot touch.
This is the key insight behind random forests: once you have enough trees, the only way to reduce variance further is to lower . Bagging alone leaves high (trees all latch onto the same dominant feature); random feature subsets at each split push down, lowering the floor. So RF attacks the term bagging can't.
Worked example
5 trees predict [cat, dog, cat, cat, dog] for a new image. Majority vote: cat (3 vs 2).
# The idiom (for your own X_train / y_train / X_test / y_test):
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
bag = BaggingClassifier(
estimator=DecisionTreeClassifier(max_depth=None),
n_estimators=100,
max_samples=0.8, # each tree sees 80% of data
max_features=1.0, # all features
bootstrap=True,
random_state=42
)
bag.fit(X_train, y_train)
print(f"Bagging accuracy: {bag.score(X_test, y_test):.3f}")
Random Forests
Random Forests improve on bagging by adding feature randomization: at each split, only consider a random subset of features.
This decorrelates the trees further. They can't all rely on the same dominant feature:
| Task | Default |
|---|---|
| Classification | |
| Regression |
where are the random feature subsets for tree .
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(
n_estimators=100, # number of trees
max_depth=10, # limit tree depth
max_features='sqrt', # √d features per split
min_samples_leaf=5, # minimum samples in leaf
random_state=42,
n_jobs=-1 # use all CPU cores
)
rf.fit(X_train, y_train)
print(f"RF accuracy: {rf.score(X_test, y_test):.3f}")
Feature importance
Two main methods:
Mean Decrease in Impurity (MDI)
Sum the impurity decrease at each split across all trees, weighted by the number of samples reaching that split.
import numpy as np
importances = rf.feature_importances_
for i in np.argsort(importances)[::-1][:5]:
print(f"Feature {i}: {importances[i]:.3f}")
Permutation importance (more reliable)
Shuffle each feature and measure accuracy drop:
from sklearn.inspection import permutation_importance
result = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=42)
for i in np.argsort(result.importances_mean)[::-1][:5]:
print(f"Feature {i}: {result.importances_mean[i]:.3f} ± {result.importances_std[i]:.3f}")
MDI vs Permutation importance
MDI can be biased toward high-cardinality features (e.g., ID numbers). Permutation importance is more reliable but slower. Use permutation importance for final analysis, MDI for quick exploration.
Out-of-Bag (OOB) evaluation
Where does "~37%" come from? A bootstrap sample draws rows with replacement. The chance a specific row is not picked on one draw is , and draws are independent, so the chance it's missed by all draws is:
So each tree leaves out about 37% of the rows (e.g. gives ). Those out-of-bag rows were never seen by that tree, so we can use them as a built-in validation set — no cross-validation needed:
rf = RandomForestClassifier(n_estimators=100, oob_score=True, random_state=42)
rf.fit(X_train, y_train)
print(f"OOB accuracy: {rf.oob_score_:.3f}") # ~same as test accuracy
Watch it run
The trace fits real CART regression trees over 50 independent training sets, so the variance formula above can be checked rather than restated.
Bagging — the variance floor, and why random forests raise σ² on purpose
One bootstrap sample: draw 160 row indices **with replacement**, so some rows arrive several times and others not at all. This draw left 61 of 160 rows out — 38.1%. The lesson derives the expected fraction as (1 − 1/n)^n, which converges to 1/e = 0.3679; at n = 100 that is 0.3660, matching the 0.366 the page quotes. Those left-out rows are the whole basis of OOB evaluation — this tree has genuinely never seen them, so they are a free validation set.
| n | (1 − 1/n)^n |
|---|---|
| 10 | 0.3487 |
| 50 | 0.3642 |
| 100 | 0.3660 |
| 1,000 | 0.3677 |
| 10,000 | 0.3679 |
Real CART regression trees over 50 independent training sets, so the lesson's variance decomposition can be checked rather than restated. Measured ensemble variance tracks ρσ² + (1−ρ)σ²/B across a sweep of B, and the floor is visible: past about 30 trees the curve flattens onto ρσ² and more trees buy nothing. The payoff is what random forests actually trade — sampling 2 of 5 features drops correlation from 0.62 to 0.38 while *raising* per-tree variance, because a tree denied the best split is a worse estimator; the floor falls anyway. Finally the OOB estimate is checked against a held-out test set, including the reason it errs pessimistic: it evaluates a smaller ensemble than the one you ship.
What to notice:
- Frame 3: the formula holds, and the floor is visible. Measured ensemble variance tracks across a sweep of . By the second term is down to 0.02 while the first sits immovably at 0.18, so more trees buy essentially nothing.
- Frame 4: averaging does not touch the bias. Single tree and 40-tree ensemble have the same bias at the probe point; only the variance collapses. Averaging is linear, so bagging cannot fix a systematically wrong model — only an unstable one. That is the gap boosting exists to attack.
- Frame 5: what random forests actually trade. Sampling 2 of 5 features drops from 0.172 to 0.072 as promised, but per-tree variance rises from 1.05 to 1.83, because a tree denied the best split is a worse estimator. The floor falls 28% anyway. RF deliberately accepts a worse individual model to buy decorrelation that more trees can exploit.
- Frame 6: OOB against a real test set. Over 45 datasets the OOB error is +0.186 ± 0.111 above the held-out test error, about 6% — close enough to use, and biased in the conservative direction because an OOB prediction averages only the ~22 trees that excluded that row.
Strengths and weaknesses
| ✅ Strengths | ❌ Weaknesses |
|---|---|
| Robust to overfitting | Less interpretable than single tree |
| Handles high-dimensional data | Can be slow for large datasets |
| Parallelizable | Cannot extrapolate (interpolation only) |
| Provides feature importance | Struggles with very sparse data |
| Minimal hyperparameter tuning | Black-box compared to single tree |
| Handles missing values |
Random Forests are the best tabular baseline
If you have structured/tabular data and don't know where to start, try Random Forests first. They require minimal tuning, are robust to overfitting, and provide feature importance. Only move to gradient boosting if you need more accuracy.
Exercise
What does bagging primarily reduce in a model?
Need a hint?
Bagging averages multiple models trained on different subsets.
Exercise
How does permutation importance measure feature importance?
Need a hint?
It measures what happens when you shuffle a feature.
Exercise
What percentage of training samples are left out of each bootstrap sample in bagging?
Need a hint?
Each bootstrap sample draws n samples with replacement from n total.
Common mistakes
- Leaving trees completely unconstrained: cap complexity one way or the other: either
max_depth=10, or if you keepmax_depth=None, setmin_samples_leaf=5so leaves can't chase single points. - Using too few trees: 10 trees is not enough. Use 100-500. More trees rarely hurt (diminishing returns), but few trees give noisy estimates.
- Ignoring feature scaling: Random Forests don't need scaling! They're invariant to monotonic transformations. Don't waste time preprocessing.
Key takeaways
- Bagging trains models on bootstrap samples and averages them, cutting variance.
- Random forests add per-split feature randomness to decorrelate the trees.
- Out-of-bag rows give a built-in validation score.
- Prefer permutation importance over impurity-based importance for reliability.
Related concepts
- Boosting: the sequential, bias-reducing ensemble
- Decision Trees: the base learner being averaged