Skip to content
ML to LLMs
Labs / Classical ML

Gradient boosting, round by round

Watch each stump fit the residuals; tune learning rate and rounds and find where test error turns up.

Theory and worked examples: Random forests and gradient boosting

Gradient boosting, round by round

Synthetic 1D regression (y = sin x + 0.3x + noise). Each round fits a one-split tree (a stump) to the current residuals and adds a fraction of it.

Data, ensemble prediction, and the next tree0246810-2024xy
F5(x)after adding the next treeresiduals
Error versus number of trees05010015020000.511.5roundsMSE
traintestcurrent round

Next update: F6(x) = F5(x) + 0.30 × (-0.24 if x < 5.65 else 0.20). The stump (purple) is fitted to the residuals (gray bars), not to y.

Each round removes part of what's left. The smaller η is, the smaller and more careful each correction, and the more rounds it takes.