Probability and statistics

1. Intuition

Conditional probability is probability after you learn something. The chance a random email is spam might be 20%; the chance it is spam given that it contains "free money" is much higher.

Bayes' rule flips a conditional: from "how often does a positive test happen when someone is sick" to "how likely is someone sick given a positive test." The second depends heavily on how common the disease is in the first place (the base rate). A 95%-accurate test for a rare disease produces mostly false alarms.

Expectation is the long-run average; variance measures spread around it.

Correlation means two quantities move together. It does not tell you that one causes the other: ice-cream sales and swimming incidents both rise in summer because of temperature, not because of each other.

2. Visualization

Right panel is synthetic: both variables are generated from temperature and do not affect each other, yet their correlation is 0.66.

3. The math

Symbols

Core rules

The middle one is Bayes' rule: posterior = likelihood × prior / evidence.

Distributions you will meet

Worked example 1: Bayes' rule and the base rate

A disease affects 1% of people. A test detects it 95% of the time (sensitivity) and gives a false positive 5% of the time. You test positive. What is the chance you have the disease?

Only 16.1%. Out of 10,000 people, 95 sick people test positive and 495 healthy people test positive. The false positives swamp the true ones because healthy people are 99 times more common. This is exactly why accuracy misleads on imbalanced classification.

Worked example 2: expectation and variance

A die roll $X$: $\mathbb{E}[X] = (1+2+3+4+5+6)/6 = 3.5$. $\mathbb{E}[X^2] = (1+4+9+16+25+36)/6 = 91/6 = 15.17$, so $\text{Var}(X) = \mathbb{E}[X^2] - \mathbb{E}[X]^2 = 15.17 - 12.25 = 2.92$ and $\sigma = 1.71$.

Worked example 3: sampling error

A classifier gets 870 of 1,000 test examples right: accuracy $\hat{p} = 0.87$. The standard error of a proportion is $\sqrt{\hat{p}(1-\hat{p})/n} = \sqrt{0.87 \times 0.13 / 1000} = \sqrt{0.0001131} = 0.0106$. A rough 95% interval is $0.87 \pm 1.96 \times 0.0106 = [0.849, 0.891]$. A competing model scoring 0.88 on the same set is not clearly better.

4. Implementation

You will build the statistics that ML evaluation leans on, using only the Python standard library: variance, correlation, Bayes' rule, a bootstrap confidence interval, and a permutation test for "is model A really better than model B?". Each one reproduces a worked example above.

Build it from scratch

Mean and variance first. The sample=True branch divides by n − 1 (Bessel's correction): a sample's points sit closer to their own mean than to the true mean, so dividing by n underestimates the spread.

def mean(xs):
    return sum(xs) / len(xs)

def variance(xs, sample=True):
    m = mean(xs)
    ss = sum((x - m) ** 2 for x in xs)
    return ss / (len(xs) - 1 if sample else len(xs))   # n - 1: Bessel's correction

Predict: for [2, 4, 4, 4, 5, 5, 7, 9] (mean 5, squared deviations summing to 32), what are the population and sample variances?

Pearson correlation is covariance divided by both spreads, so it lands in [−1, 1] whatever the units.

def pearson(x, y):
    mx, my = mean(x), mean(y)
    cov = sum((a - mx) * (b - my) for a, b in zip(x, y))
    return cov / (sum((a - mx) ** 2 for a in x) * sum((b - my) ** 2 for b in y)) ** 0.5

Bayes' rule is worked example 1 as a function. The denominator is the total probability of a positive test, from both sick and healthy people.

def bayes(prior, sensitivity, false_positive_rate):
    evidence = sensitivity * prior + false_positive_rate * (1 - prior)   # P(positive)
    return sensitivity * prior / evidence

Predict: raise the prior from 1% to 10% and keep the test the same. Does P(disease | positive) roughly double, or more?

The bootstrap estimates uncertainty without a formula: resample the data with replacement many times, recompute the statistic each time, and read off the middle 95% of the results.

def bootstrap_ci(xs, stat=mean, n_boot=2000, alpha=0.05, rng=random):
    boots = sorted(stat(rng.choices(xs, k=len(xs))) for _ in range(n_boot))
    return boots[int(n_boot * alpha / 2)], boots[int(n_boot * (1 - alpha / 2)) - 1]

A permutation test asks: if the labels "A" and "B" meant nothing, how often would shuffling them produce a gap at least this large? That fraction is the p-value. The +1 on top and bottom keeps it from ever being exactly 0.

def permutation_test(a, b, n_perm=5000, rng=random):
    """Two-sided p-value for 'mean(a) - mean(b) is zero': shuffle the labels and count."""
    observed = abs(mean(a) - mean(b))
    pooled = a + b
    hits = 0
    for _ in range(n_perm):
        rng.shuffle(pooled)
        if abs(mean(pooled[:len(a)]) - mean(pooled[len(a):])) >= observed:
            hits += 1
    return (hits + 1) / (n_perm + 1)

Predict: you run the bootstrap on 870 correct out of 1,000. Should the interval match worked example 3's [0.849, 0.891]? Would it get narrower with 10,000 resamples, or with 10,000 test examples?

Use the library

The standard library already has these: statistics.mean, statistics.variance (sample, n − 1), statistics.pvariance (population), and statistics.correlation. For tests and intervals, SciPy is the usual tool:

from scipy import stats

stats.ttest_ind(a, b, equal_var=False)                      # Welch's t-test
stats.permutation_test((a, b), lambda x, y: x.mean() - y.mean(), n_resamples=5000)
stats.bootstrap((correct,), np.mean, confidence_level=0.95)

In NumPy, watch the default: np.var and np.std divide by n (ddof=0), while pandas .var() and .std() divide by n − 1. The scratch code made that choice explicit; libraries make it silently. (SciPy is not in this repo's dependencies; the script uses only the standard library.)

Run it

From guide/:

uv run code/00-foundations/stats_scratch.py

mean 5.0, population var 4.0, sample var 4.5714
P(disease | positive) = 0.1610
CLT: std of 5000 sample means 0.3096, predicted σ/√30 = 0.3118
accuracy 0.870, bootstrap 95% CI [0.850, 0.890]
mean A 0.810, mean B 0.753, permutation p = 0.0040
r(ice cream, incidents) = 0.552  (confounder: temperature)
all checks match the statistics module

The CLT line checks worked example 2: a die has σ = 1.708, so the mean of 30 rolls should have spread 1.708/√30 = 0.312. All data is synthetic and seeded. Full script: code/00-foundations/stats_scratch.py. Figures: code/00-foundations/foundations.py.

5. Engineering

Every metric is an estimate. Report confidence intervals (bootstrap is simple and general), especially for small test sets or rare classes. Compare models on the same examples (paired comparisons are far more sensitive).

Base rates drive precision. When positives are rare, even a good classifier's alarms are mostly false. Choose thresholds and metrics with the base rate in mind.

Correlation versus causation. A predictive model exploits correlations; it doesn't learn what would happen if you intervened. Don't read coefficients or feature importances as causal effects. Estimating causal effects needs experiments (A/B tests) or causal-inference methods.

Sampling and representativeness. A model learns the distribution it was trained on. If production users differ (new regions, seasons, products), performance changes. That is distribution shift.

Watch out

Failure modes. Ignoring base rates; treating a 0.5-point metric difference on 500 examples as real; drawing causal conclusions from observational data; training on a sample that doesn't represent production.

Common mistakes

• Confusing $P(A \mid B)$ with $P(B \mid A)$ (the prosecutor's fallacy).
• Assuming independence when events are linked.
• Reporting a single number without a sense of its uncertainty.

6. Knowledge check

Practice exercise. 30% of emails are spam. "Winner" appears in 40% of spam and 2% of non-spam. What is $P(\text{spam} \mid \text{"winner"})$?

Implementation challenge. Simulate the disease example for base rates of 0.1%, 1%, 10%, and 50%, and plot $P(D \mid +)$. Explain the curve.

Summary

• Conditional probability updates beliefs with evidence; Bayes' rule flips the conditional and depends on the prior.
• Rare positives make even accurate tests produce mostly false alarms.
• Expectation is the long-run mean; variance is spread. Know Bernoulli, binomial, categorical, and normal.
• Every evaluation metric is an estimate with uncertainty.
• Correlation is not causation; predictive models capture correlations.

Next: The Python toolkit

Related: Naive Bayes · Classification metrics

Go deeper, code-first (AI Engineering from Scratch): Sampling Methods · Stochastic Processes

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Probability and Distributions, Statistics for Machine Learning.

Interview angle