Cheat sheets
Regularization
Every technique that fights overfitting, what it does, and when to reach for it.
| Technique | What it does | Where |
|---|---|---|
| L2 (ridge, weight decay) | adds ; shrinks weights smoothly | linear models, neural nets (AdamW) |
| L1 (lasso) | adds ; zeroes weights | sparse linear models |
| Elastic net | L1 + L2 | correlated features with sparsity |
| Early stopping | keep the best validation checkpoint | boosting, neural nets |
| Dropout | randomly zero activations (scale by ) | neural nets |
| Data augmentation | label-preserving variations | vision, audio, text |
| Tree constraints | max depth, min samples per leaf, pruning | trees and ensembles |
| Shrinkage | small learning rate per boosting round | gradient boosting |
| Bagging | average models on bootstrap samples | random forests |
| Normalization | batch/layer norm stabilizes activations | deep nets |
| Smaller model / fewer features | reduce capacity | anywhere |
| More data | the most reliable fix for variance | anywhere |
Diagnose first: a large train-validation gap means overfitting (regularize); both errors high means underfitting (add capacity, features, or train longer).
Always: scale features before L1/L2; tune strength by cross-validation on a log scale; never tune on the test set.
Lessons: Overfitting and bias-variance · Regularization for linear models · Training and regularization