Gradient Boosting in Practice: XGBoost
Boosting gave us the recipe: fit each new tree to the gradient of the loss, nudging the ensemble downhill in function space. XGBoost (and its cousins LightGBM and CatBoost) turned that recipe into the engine that has dominated tabular-data machine learning for a decade. The difference is regularization — applied everywhere.
The regularized objective
XGBoost doesn't just fit the gradient; it minimizes a penalized objective at every tree:
where is the number of leaves in a tree and are its leaf output values. Two penalties, two effects:
- — each leaf must justify its existence: a split happens only if it improves the loss by more than . Built-in pruning.
- — L2 shrinkage on leaf values, the same medicine as Ridge regression, applied inside every tree.
XGBoost also uses the second-order Taylor expansion of the loss (gradients and Hessians), which gives a closed-form optimal leaf weight and split score instead of a heuristic.
Shrinkage: the learning rate
Each tree's contribution is scaled by a learning rate :
Small (0.01–0.1) means each tree corrects only a fraction of the remaining error, leaving room for later trees to refine. This is the single most reliable quality knob: lower + more trees almost always generalizes better, at the cost of training time. The standard move is to fix small and use early stopping — keep adding trees until validation error stops improving.
Try the trade yourself — drag η down and watch how many more trees it takes to fit, but how much later (and lower) the test error bottoms out:
Shrinkage: many small steps beat few big ones
Top: the boosted fit (teal) built from depth-1 stumps vs the truth (dashed). Bottom: train (indigo) and test (rose) error as trees are added — the dotted marker is the test-error minimum. Lower η with more trees reaches a better test minimum and overfits later; η = 1 races down the train curve and overfits early.
Watch a tree get built
Everything above — the second-order gain, the closed-form leaf weight, , and — is one loop over eight points. Step through it:
XGBoost's split search — gradients, Hessians, λ and γ
Eight points, one feature, binary labels — and two of them (x = 3 and x = 4) sit on the wrong side of any single threshold, so no stump can be perfect. Boosting starts from F = 0, which is log-odds 0, so every prediction is p = 0.5. From that, g = p − y is ±0.5 and h = p(1−p) is **0.25 for every sample**. That uniformity is temporary and worth remembering: at round 1 the Hessians carry no information at all, and everything the second-order machinery buys shows up later.
| x | y | p | g = p−y | h = p(1−p) |
|---|---|---|---|---|
| 1 | 0 | 0.5000 | 0.5000 | 0.2500 |
| 2 | 0 | 0.5000 | 0.5000 | 0.2500 |
| 3 | 1 | 0.5000 | −0.5000 | 0.2500 |
| 4 | 0 | 0.5000 | 0.5000 | 0.2500 |
| 5 | 1 | 0.5000 | −0.5000 | 0.2500 |
| 6 | 1 | 0.5000 | −0.5000 | 0.2500 |
| 7 | 1 | 0.5000 | −0.5000 | 0.2500 |
| 8 | 1 | 0.5000 | −0.5000 | 0.2500 |
The regularized objective from this lesson, executed on eight points. Each round recomputes g = p − y and h = p(1−p), scores every candidate threshold with the closed-form second-order gain, and adds η times the optimal leaf weight −G/(H+λ). Frame 5 is the contrast with the AdaBoost trace: no sample weight is ever touched — a badly-fit point's gradient simply grows, and the split score is built from gradients. The last two frames measure what the two regularizers actually do, and they do different jobs: λ shrinks leaf values (and quietly refuses to isolate single points), while γ has a hard threshold above which no split clears the bar and the entire ensemble becomes a no-op.
What to notice:
- The split score is not a heuristic. Unlike information gain in a plain decision tree, is the exact loss reduction under the second-order expansion, which is why the Hessians are needed. The leaf value is closed-form too.
- Gradient boosting never re-weights a sample. Compare AdaBoost, which multiplies the weight of every misclassified point. Here nothing is weighted: a badly-fit point's gradient simply is larger next round, and the split score is built from gradients, so the next tree goes after it on its own.
- does more than shrink leaves. One split scores negative at
before is even applied, because it would isolate a single
point in a leaf with a tiny Hessian — the same conservatism
min_child_weightenforces explicitly. - The Hessians only start mattering later. At round 1 every ; by round 40 they span nearly 5×, and the confidently-predicted points have and drop out of the split score entirely.
- has a hard threshold you can compute. The best gain available in round 1 is 1.083, so at no split ever clears the bar, no tree is ever built, and 50 rounds of boosting leave the loss at exactly . The ensemble is a silent no-op.
Row and column subsampling
Borrowing from Random Forests, each tree (or each split) sees only a random fraction of rows (subsample) and columns (colsample_bytree). Decorrelating the trees reduces variance, and as a bonus makes training faster.
A sane starting configuration
eta=0.05, max_depth=6, subsample=0.8, colsample_bytree=0.8, n_estimators large with early_stopping_rounds=50. Then tune max_depth (capacity), min_child_weight (leaf conservatism), and the regularizers lambda/gamma — in that order. Resist tuning ten knobs at once.
That configuration, in code (for your own train/test split — XGBoost and LightGBM share the same shape):
import xgboost as xgb
import lightgbm as lgb
xgb_model = xgb.XGBClassifier(
n_estimators=500,
max_depth=6,
learning_rate=0.05,
subsample=0.8,
colsample_bytree=0.8,
eval_metric='logloss',
early_stopping_rounds=50,
)
xgb_model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)
lgb_model = lgb.LGBMClassifier(
n_estimators=500,
max_depth=6,
learning_rate=0.05,
subsample=0.8,
colsample_bytree=0.8,
early_stopping_round=50,
)
lgb_model.fit(X_train, y_train, eval_set=[(X_test, y_test)])
Why trees still beat neural nets on tables
On heterogeneous tabular data (mixed scales, missing values, categorical features, low sample counts), gradient-boosted trees remain the strongest default:
- Splits are invariant to feature scaling and monotone transforms; no normalization needed
- XGBoost handles missing values natively, learning a default direction per split
- Trees capture sharp interactions and thresholds that smooth function approximators need many samples to find
Deep learning wins when there's structure to exploit — images, text, audio, huge datasets. For a 50,000-row CSV, boosted trees are the bar to beat.
Exercise
You cut XGBoost's learning rate η from 0.3 to 0.03. What else must change for good results?
Need a hint?
Each tree now corrects 10× less of the remaining error.
Exercise
Which XGBoost hyperparameter is most important to tune first?
Need a hint?
Think about the bias-variance tradeoff.
Key takeaways
- XGBoost = gradient boosting + regularization at every level: penalized leaves (, ), shrinkage (), and row/column subsampling
- Second-order (gradient + Hessian) approximation gives principled split scores and leaf values
- Lower learning rate + more trees + early stopping is the most dependable tuning move
- For tabular data, gradient-boosted trees are still the default champion
Related concepts
- Boosting: the algorithm this system industrializes
- Bagging & Random Forests: where the row/column subsampling idea comes from
- Hyperparameter Optimization: searching the knobs beyond the sane defaults