Learning path
Next moduleModule 23: Projects and career roadmap
Module 22
ML system design and experimentation
Design an ML system end to end with a repeatable framework, size a recommendation funnel against a latency budget, and prove it works with a well-powered A/B test.
- 1. ML system designUse a repeatable ten-step framework to design an ML system in an interview or at work, and apply it end to end to a multi-stage recommendation feed with a latency budget.50 min
- 2. Experimentation and A/B testingDesign an A/B test that can actually detect the effect you care about, read its result correctly, and avoid the traps (peeking, sample ratio mismatch, novelty, interference) that make online experiments lie.45 min
- 3. Model monitoring and driftMonitor a deployed model across system, input, prediction, and outcome layers, tell covariate, prior, and concept drift apart, compute PSI, the KS statistic, and chi-square from scratch, and design an alert policy that catches real drift without paging people for noise.45 min
A model is one box in a system. This module covers the rest: framing the problem, choosing metrics, building labels and features without leakage, serving within a latency budget, and deciding with an online experiment whether the new system is actually better.
| Step | What you decide | Failure mode |
|---|---|---|
| Clarify and frame | goal, constraints, prediction target | optimizing the wrong thing |
| Metrics | business, online, offline, guardrails | offline wins that lose online |
| Data and features | label source, point-in-time features, feature store | leakage, training-serving skew |
| Model and serving | baseline, funnel stages, latency budget | blowing the budget, no fallback |
| Experiment | sample size, duration, randomization unit | peeking, SRM, interference |
| Monitor | drift, calibration, freshness | silent decay, feedback loops |
| # | Lesson | You will be able to |
|---|---|---|
| 1 | ML system design | run a ten-step design framework and design a multi-stage recommendation funnel with a latency budget |
| 2 | Experimentation and A/B testing | compute sample size and power, read a z-test, and catch peeking, SRM, novelty, and interference |
| 3 | Model monitoring and drift | tell covariate, prior, and concept drift apart, compute PSI, KS, and chi-square, and design an alert policy that avoids alert fatigue |