Cheat sheets
Model selection
What to try first for each kind of problem, what to try next, and the questions to ask before trusting any result.
Start simple, then earn complexity
Always fit a baseline first (predict the mean, the majority class, or a one-feature rule). A complex model only earns its place if it beats the baseline on a validation set that looks like production.
| Situation | Start with | Then consider | Lessons |
|---|---|---|---|
| Continuous tabular target | Linear / ridge regression | Random forest, gradient boosting | Linear regression, Ensembles |
| Binary or multiclass tabular | Logistic regression | XGBoost, LightGBM, CatBoost | Logistic regression |
| Very small labeled dataset | Logistic regression, Naive Bayes, KNN | SVM, regularized linear models | Naive Bayes |
| Sparse text with labels | TF-IDF + linear model | Fine-tuned transformer encoder | Text to vectors |
| No labels, want groups | K-means | DBSCAN, hierarchical, Gaussian mixture | K-means |
| Rare events, few labels | Isolation-style anomaly scoring, thresholds | Semi-supervised, density models | Clustering |
| High-dimensional visualization | PCA | UMAP / t-SNE (exploration only) | PCA |
| Image classification | Pretrained CNN or vision transformer | Fine-tuning, data augmentation | CNNs |
| Sequences / time series | Lagged features + gradient boosting | RNN/LSTM, temporal transformers | RNNs |
| Text generation | Pretrained decoder LLM + prompting | RAG, LoRA / PEFT, fine-tuning | Adapting LLMs |
| Q&A over private or fresh knowledge | Retrieval + grounded generation | Hybrid search, reranking | RAG |
| Strict low latency | Simple model or small encoder | Distillation, quantization, caching | Reliability and cost |
Engineering noteFor structured (tabular) data, gradient-boosted trees are usually the strongest baseline. For raw images, audio, and language, deep learning has the better inductive bias.
Choosing by constraint
| If you need... | Prefer | Avoid |
|---|---|---|
| Explanations stakeholders can read | Linear / logistic regression, shallow trees | Deep ensembles without explanation tooling |
| Calibrated probabilities | Logistic regression, calibrated boosting | Raw SVM scores, uncalibrated trees |
| Robustness to unscaled features | Trees and tree ensembles | KNN, SVM, neural nets, K-means without scaling |
| Training on tiny data | Strong priors: Naive Bayes, regularized linear | Large neural networks from scratch |
| Fast inference on CPU | Linear models, small trees, distilled models | KNN over large datasets, large transformers |
| Frequently changing facts | RAG (change the context) | Fine-tuning (changes weights, slow to update) |
| A new output format or style | Prompting, then fine-tuning / LoRA | RAG alone |
How to study any model
- State the task and output type.
- Draw its input-to-output computation.
- Write its objective or loss.
- List assumptions and required preprocessing.
- Identify the key hyperparameters.
- Choose suitable evaluation metrics.
- Describe failure modes.
- Implement a baseline.
- Do error analysis and compare against a simpler model.
Questions to ask before trusting a result
- Was the test set truly unseen and representative of production?
- Could any feature leak the target or future information?
- Is the metric aligned with the business cost of errors and with class imbalance?
- How does the model perform across meaningful slices (segments, regions, time)?
- Is it calibrated? Does confidence match observed accuracy?
- What happens under missing values, drift, adversarial inputs, and outages?
- Can the pipeline be reproduced from the data version, code version, configuration, and random seed?
See every model side by side in the model cards.