Ensemble Methods Quiz
Check your understanding of ensembles, which error component each method attacks, what makes Random Forests more than bagged trees, and what gradient boosting fits at each round.
Exercise
Bagging primarily reduces which component of error?
Need a hint?
Averaging many noisy estimates does what to the noise?
Exercise
How does boosting differ fundamentally from bagging?
Need a hint?
Can you train boosted trees in parallel?
Exercise
Beyond bagging, what extra randomness does a Random Forest add?
Need a hint?
Look at what each split is allowed to consider.
Exercise
What are out-of-bag (OOB) samples used for?
Need a hint?
Each bootstrap sample leaves out ~37% of the training points.
Exercise
In gradient boosting, what does each new tree fit?
Need a hint?
For squared error, this is just the residuals.
Exercise
Gradient boosting adds each tree by taking a gradient step in function space. Which slot of the ML project loop is that fundamentally an instance of?
Need a hint?
It's a search that drives the ensemble downhill on the loss, one step per round.
Exercise
Transfer test — a technique this course did NOT teach: dropout randomly zeroes activations during training so a big network can't memorize. Which slot does it modify?
Need a hint?
It restricts the effective capacity of the model family.