Gradient Boosting in Practice: XGBoostOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Gradient Boosting in Practice: XGBoost

Boosting gave us the recipe: fit each new tree to the gradient of the loss, nudging the ensemble downhill in function space. XGBoost (and its cousins LightGBM and CatBoost) turned that recipe into the engine that has dominated tabular-data machine learning for a decade. The difference is regularization — applied everywhere.

The regularized objective

XGBoost doesn't just fit the gradient; it minimizes a penalized objective at every tree:

L=∑iℓ(yi,y^i)  +  ∑trees[γT+12λ∑j=1Twj2]\mathcal{L} = \sum_{i} \ell(y_i, \hat{y}_i) \;+\; \sum_{\text{trees}} \left[ \gamma T + \frac{1}{2}\lambda \sum_{j=1}^{T} w_j^2 \right]

where TT is the number of leaves in a tree and wjw_j are its leaf output values. Two penalties, two effects:

  • γT\gamma T — each leaf must justify its existence: a split happens only if it improves the loss by more than γ\gamma. Built-in pruning.
  • λ∑wj2\lambda \sum w_j^2 — L2 shrinkage on leaf values, the same medicine as Ridge regression, applied inside every tree.

XGBoost also uses the second-order Taylor expansion of the loss (gradients and Hessians), which gives a closed-form optimal leaf weight and split score instead of a heuristic.

Shrinkage: the learning rate

Each tree's contribution is scaled by a learning rate η\eta:

y^(t)=y^(t−1)+η ft(x)\hat{y}^{(t)} = \hat{y}^{(t-1)} + \eta \, f_t(x)

Small η\eta (0.01–0.1) means each tree corrects only a fraction of the remaining error, leaving room for later trees to refine. This is the single most reliable quality knob: lower η\eta + more trees almost always generalizes better, at the cost of training time. The standard move is to fix η\eta small and use early stopping — keep adding trees until validation error stops improving.

Try the trade yourself — drag η down and watch how many more trees it takes to fit, but how much later (and lower) the test error bottoms out:

Shrinkage: many small steps beat few big ones

xy00.40.81255075100MSE
— boosted fit- - true function— train MSE— test MSE
train MSE0.105
test MSE0.176
best test @30 trees

Top: the boosted fit (teal) built from depth-1 stumps vs the truth (dashed). Bottom: train (indigo) and test (rose) error as trees are added — the dotted marker is the test-error minimum. Lower η with more trees reaches a better test minimum and overfits later; η = 1 races down the train curve and overfits early.

Watch a tree get built

Everything above — the second-order gain, the closed-form leaf weight, λ\lambda, γ\gamma and η\eta — is one loop over eight points. Step through it:

XGBoost's split search — gradients, Hessians, λ and γ

1/9

Eight points, one feature, binary labels — and two of them (x = 3 and x = 4) sit on the wrong side of any single threshold, so no stump can be perfect. Boosting starts from F = 0, which is log-odds 0, so every prediction is p = 0.5. From that, g = p − y is ±0.5 and h = p(1−p) is **0.25 for every sample**. That uniformity is temporary and worth remembering: at round 1 the Hessians carry no information at all, and everything the second-order machinery buys shows up later.

python
1def gain(GL, HL, GR, HR, lam, gam):
2 s = lambda G, H: G*G / (H + lam)
3 parent = s(GL + GR, HL + HR)
4 return 0.5 * (s(GL, HL) + s(GR, HR)
5 - parent) - gam
6
7def best_split(X, g, h, lam, gam):
8 G, H = g.sum(), h.sum()
9 top = (0.0, None)
10 for t in midpoints(X):
11 L = X < t
12 GL, HL = g[L].sum(), h[L].sum()
13 v = gain(GL, HL, G - GL,
14 H - HL, lam, gam)
15 if v > top[0]:
16 top = (v, t)
17 return top
18
19def boost(X, y, n, eta, lam, gam):
20 F = np.zeros(len(y))
21 for _ in range(n):
22 p = 1 / (1 + np.exp(-F))
23 g, h = p - y, p * (1 - p)
24 v, t = best_split(X, g, h,
25 lam, gam)
26 # no split cleared gam: no tree
27 if t is None:
28 continue
29 # leaf weight w = -G / (H + lam)
30 w = leaves(X, t, g, h, lam)
31 F += eta * w
32 return F
state
round 1 — gradients and Hessians
xypg = p−yh = p(1−p)
100.50000.50000.2500
200.50000.50000.2500
310.5000−0.50000.2500
400.50000.50000.2500
510.5000−0.50000.2500
610.5000−0.50000.2500
710.5000−0.50000.2500
810.5000−0.50000.2500
start
loss → 0.693147ln 2 → 0.693147λ → 1γ → 0η → 0.3

The regularized objective from this lesson, executed on eight points. Each round recomputes g = p − y and h = p(1−p), scores every candidate threshold with the closed-form second-order gain, and adds η times the optimal leaf weight −G/(H+λ). Frame 5 is the contrast with the AdaBoost trace: no sample weight is ever touched — a badly-fit point's gradient simply grows, and the split score is built from gradients. The last two frames measure what the two regularizers actually do, and they do different jobs: λ shrinks leaf values (and quietly refuses to isolate single points), while γ has a hard threshold above which no split clears the bar and the entire ensemble becomes a no-op.

What to notice:

  • The split score is not a heuristic. Unlike information gain in a plain decision tree, 12[GL2HL+λ+GR2HR+λ−G2H+λ]−γ\frac{1}{2}\left[\frac{G_L^2}{H_L+\lambda} + \frac{G_R^2}{H_R+\lambda} - \frac{G^2}{H+\lambda}\right] - \gamma is the exact loss reduction under the second-order expansion, which is why the Hessians are needed. The leaf value w∗=−G/(H+λ)w^* = -G/(H+\lambda) is closed-form too.
  • Gradient boosting never re-weights a sample. Compare AdaBoost, which multiplies the weight of every misclassified point. Here nothing is weighted: a badly-fit point's gradient simply is larger next round, and the split score is built from gradients, so the next tree goes after it on its own.
  • λ\lambda does more than shrink leaves. One split scores negative at λ=1\lambda = 1 before γ\gamma is even applied, because it would isolate a single point in a leaf with a tiny Hessian — the same conservatism min_child_weight enforces explicitly.
  • The Hessians only start mattering later. At round 1 every h=0.25h = 0.25; by round 40 they span nearly 5×, and the confidently-predicted points have h→0h \to 0 and drop out of the split score entirely.
  • γ\gamma has a hard threshold you can compute. The best gain available in round 1 is 1.083, so at γ=1.09\gamma = 1.09 no split ever clears the bar, no tree is ever built, and 50 rounds of boosting leave the loss at exactly ln⁡2\ln 2. The ensemble is a silent no-op.

Row and column subsampling

Borrowing from Random Forests, each tree (or each split) sees only a random fraction of rows (subsample) and columns (colsample_bytree). Decorrelating the trees reduces variance, and as a bonus makes training faster.

A sane starting configuration

eta=0.05, max_depth=6, subsample=0.8, colsample_bytree=0.8, n_estimators large with early_stopping_rounds=50. Then tune max_depth (capacity), min_child_weight (leaf conservatism), and the regularizers lambda/gamma — in that order. Resist tuning ten knobs at once.

That configuration, in code (for your own train/test split — XGBoost and LightGBM share the same shape):

Python
import xgboost as xgb
import lightgbm as lgb

xgb_model = xgb.XGBClassifier(
    n_estimators=500,
    max_depth=6,
    learning_rate=0.05,
    subsample=0.8,
    colsample_bytree=0.8,
    eval_metric='logloss',
    early_stopping_rounds=50,
)
xgb_model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)

lgb_model = lgb.LGBMClassifier(
    n_estimators=500,
    max_depth=6,
    learning_rate=0.05,
    subsample=0.8,
    colsample_bytree=0.8,
    early_stopping_round=50,
)
lgb_model.fit(X_train, y_train, eval_set=[(X_test, y_test)])

Why trees still beat neural nets on tables

On heterogeneous tabular data (mixed scales, missing values, categorical features, low sample counts), gradient-boosted trees remain the strongest default:

  • Splits are invariant to feature scaling and monotone transforms; no normalization needed
  • XGBoost handles missing values natively, learning a default direction per split
  • Trees capture sharp interactions and thresholds that smooth function approximators need many samples to find

Deep learning wins when there's structure to exploit — images, text, audio, huge datasets. For a 50,000-row CSV, boosted trees are the bar to beat.

Exercise

You cut XGBoost's learning rate η from 0.3 to 0.03. What else must change for good results?

Need a hint?

Each tree now corrects 10× less of the remaining error.

Exercise

Which XGBoost hyperparameter is most important to tune first?

Need a hint?

Think about the bias-variance tradeoff.

Key takeaways

  • XGBoost = gradient boosting + regularization at every level: penalized leaves (γ\gamma, λ\lambda), shrinkage (η\eta), and row/column subsampling
  • Second-order (gradient + Hessian) approximation gives principled split scores and leaf values
  • Lower learning rate + more trees + early stopping is the most dependable tuning move
  • For tabular data, gradient-boosted trees are still the default champion

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Gradient Boosting in Practice: XG…KNN & Decision TreesBoostingBagging & Random ForestsHyperparameter Optimizati…