Learning path
Next moduleModule 7: Unsupervised learning
Module 6
Trees and ensembles
Decision trees, random forests, and gradient boosting, the strongest default family for tabular data.
- 1. Decision treesSplit data with simple rules, choose splits with Gini impurity or entropy by hand, and control depth to avoid memorizing the training set.30 min
- 2. Random forests and gradient boostingCombine many trees by bagging (random forests) or boosting (XGBoost, LightGBM, CatBoost), run boosting rounds by hand, and tune the knobs that matter.40 min
Trees split data with simple rules and capture nonlinear interactions without feature scaling. Alone they overfit; combined they are among the most accurate models for structured data. Bagging (random forests) reduces variance; boosting (XGBoost, LightGBM, CatBoost) reduces bias by fitting each new tree to the remaining errors.
| # | Lesson | You will be able to |
|---|---|---|
| 1 | Decision trees | compute Gini impurity and information gain and choose a split by hand |
| 2 | Random forests and gradient boosting | compare bagging and boosting and run one boosting round by hand |
Engineering noteFor structured/tabular data, gradient-boosted trees are often a top baseline. For raw images, audio, and language, deep learning usually has the stronger inductive bias.