Skip to content
ML to LLMs
Cheat sheets

ML system design

The ten-step design framework as a checklist, the recommendation funnel, data pitfalls, and the A/B test formulas.

Framework checklist (45 minutes)

  • 1. Clarify (0 to 5 min). Business goal, users and items, QPS, latency budget (p99), freshness, privacy and fairness constraints. State assumptions out loud.
  • 2. Frame. What is predicted, for whom, when. Classification, regression, ranking, or retrieval.
  • 3. Metrics. Business (retention, revenue) · online (CTR, watch time) · offline (AUC, log loss, recall@k, nDCG) · guardrails (latency, errors, diversity, safety).
  • 4. Data and labels. Source, delay, noise, selection and position bias, class balance.
  • 5. Features. User, item, context, cross. Batch versus real-time. One definition for training and serving (feature store).
  • 6. Model. Baseline first (popularity, rules, logistic regression), then GBDT, then deep models if they earn their cost.
  • 7. Training pipeline. Point-in-time joins, time-based splits, retraining cadence, versioned data and models.
  • 8. Serving. Batch or online. Latency budget per stage, caching, timeouts, fallbacks.
  • 9. Evaluate and launch. Offline gate → shadow → A/B with guardrails → staged ramp.
  • 10. Monitor and iterate. Feature freshness and nulls, prediction drift, calibration, per-segment metrics, retraining triggers.

Recommendation funnel

StageItemsModelOptimizes
Candidate generation10M → ~1,000two-tower embeddings + ANN, plus heuristic sourcesrecall, cheap per item
Ranking~1,000 → ~100GBDT or multi-task DNN on rich and cross featuresprecision at the top
Re-ranking~100 → ~20rules and slate logicdiversity, freshness, policy, ads

Latency: L=Loverhead+∑s(fs+nsts)L = L_{\text{overhead}} + \sum_s (f_s + n_s t_s). Compute: ∑snsts\sum_s n_s t_s worker-ms per request; × QPS / 1000 = busy workers.

Recalibrate after keeping a fraction ww of negatives: p=q/(q+(1−q)/w)p = q / (q + (1-q)/w). Example: q=0.5q = 0.5, w=0.1w = 0.1 gives p=0.0909p = 0.0909.

Data pitfalls

PitfallSymptomFix
Training-serving skewoffline win, online lossone feature definition; log served features and train on them
Point-in-time leakagesuspiciously high offline AUCas-of joins; time-based splits
Label leakagea feature that is a consequence of the labelaudit features against the label's timeline
Position biastop slots look great regardless of itemposition as a feature, randomization, IPW
Feedback loopdiversity collapses, new items starveexploration traffic, freshness boosts
Cold startnew users or items get poor resultscontent features, popularity fallback, exploration
Delayed labelsrecent data looks like fewer positiveswait for label maturity or model the delay

A/B testing

QuantityFormula
z-testz=(p^B−p^A)/p^(1−p^)(1/nA+1/nB)z = (\hat{p}_B - \hat{p}_A) / \sqrt{\hat{p}(1-\hat{p})(1/n_A + 1/n_B)}, p^\hat{p} pooled
Sample size per armn=(z1−α/2+z1−β)2[pA(1−pA)+pB(1−pB)]/δ2n = (z_{1-\alpha/2} + z_{1-\beta})^2 [p_A(1-p_A) + p_B(1-p_B)] / \delta^2
Shortcut (α 0.05, power 0.8)n≈16 p(1−p)/δ2n \approx 16\,p(1-p)/\delta^2
SRMχ2=∑(obs−exp)2/exp\chi^2 = \sum (\text{obs} - \text{exp})^2 / \text{exp}, 1 degree of freedom
CUPEDY−θ(X−Xˉ)Y - \theta(X - \bar{X}), variance × (1−ρ2)(1 - \rho^2)

Reference numbers: z0.975=1.960z_{0.975} = 1.960, z0.8=0.8416z_{0.8} = 0.8416. Baseline 10%, relative MDE 10%: 14,749 per arm.

Traps: peeking (fix the horizon or use sequential tests) · SRM (block the readout) · novelty and primacy (run long enough, plot the daily effect) · interference (cluster or switchback randomization) · many metrics (pre-register one primary metric).

Lessons: ML system design · Experimentation and A/B testing · Production architecture