Imbalanced data

1. Intuition

Take 800 transactions where 34 are fraud. A model that says "not fraud" every time gets $766/800 = 95.8\%$ accuracy and catches nothing. Accuracy rewards the majority class because every row counts equally, and almost every row is negative.

So first, change the scoreboard. Ask two questions about the positive class:

• Precision: of the transactions we flagged, how many were fraud?
• Recall: of the fraud that happened, how much did we flag?

F1 combines the two. MCC uses all four cells of the confusion matrix and is 0 for any constant predictor. Average precision (the area under the precision-recall curve, AUPRC) scores the whole ranking without picking a threshold, and a random ranking scores the positive rate, not 0.5.

Second, decide how the model should treat the rare class. There are three families of fixes:

1. Change the data. Duplicate positives (random oversampling), throw away negatives (random undersampling), or invent new positives between real ones (SMOTE).
2. Change the loss. Give each positive a bigger weight, so a missed fraud costs as much as roughly 19 false alarms.
3. Change the threshold. Keep the model as it is and flag anything with predicted probability above, say, 0.24 instead of 0.5.

Here is the part that surprises people. For a model like logistic regression, all three mostly do the same thing: they move the decision boundary toward the majority class. They change where you operate on the precision-recall trade-off. They barely change how well the model ranks positives above negatives. If you tune the threshold properly, the plain model often does as well as any rebalanced one.

2. Visualization

Synthetic data: two overlapping 2D Gaussian blobs, 4,000 points, 5% positive, split 2,400 / 800 / 800 into train, validation, and test. Left: class weights push the 0.5 boundary into the negatives. Middle: the best F1 threshold for the unweighted model is 0.24. Right: every rebalancing method lands on almost the same operating point, and tuning the threshold of either the plain or the weighted model reaches the same better one.

Try it (run the script, predict first, then check):

1. Change rng.normal(1.3, 0.8, ...) to rng.normal(2.5, 0.8, ...) so the classes barely overlap. Predict whether the gap between "baseline" and "class weights" in F1 grows or shrinks.
2. Change the split to n_neg=3960, n_pos=40 (1% positive). Predict what happens to the AUPRC of a random scorer and to the stability of the tuned threshold.
3. Tune the threshold on the test set instead of validation by swapping y_va, X_va for y_te, X_te in the tuning loop. Predict whether the reported test F1 goes up, and explain why that number is now untrustworthy.
4. In smote, set k=1. Predict how the teal synthetic points change in the left panel.

3. The math

Symbols

Metrics

MCC runs from $-1$ to $+1$ and is 0 for random or constant predictions. Average precision ranks rows by score and averages the precision at each true positive's rank:

where $P$ is the number of positives and $\text{rel}_k = 1$ if the row at rank $k$ is positive. A random ranking has expected AP equal to the positive rate.

Balanced class weights and weighted loss

Each class then carries the same total weight, $n_c w_c = n / C$.

SMOTE

Weights move the intercept, so they move the threshold

Up-weighting positives by the ratio $r = w_1 / w_0$ is like training on data where positives are $r$ times more common. For a logistic model this mostly adds $\ln r$ to the logit, so it multiplies the predicted odds by about $r$. To convert a weighted model's probability $p_w$ back to the real-world scale (prior correction):

For balanced weights, $r = n_0 / n_1$. The same formula maps a threshold on the weighted model to the equivalent threshold on the unweighted one. It is approximate here because the two classes have different spreads, so the true log-odds are not exactly linear.

Worked example (numbers match the script)

1. The accuracy trap. The test set has 34 positives in 800 rows. Always predicting negative gives $TN = 766$, $FN = 34$, $TP = FP = 0$. Accuracy $= 766 / 800 = 0.958$. Recall $= 0/34 = 0$, so F1 $= 0$. The MCC numerator is $0 \cdot 766 - 0 \cdot 34 = 0$, so MCC $= 0$.

2. Class weights. Training has $n = 2400$ rows, $n_1 = 121$ positives, $n_0 = 2279$ negatives, $C = 2$.

• $w_0 = 2400 / (2 \cdot 2279) = 2400 / 4558 = 0.5265$
• $w_1 = 2400 / (2 \cdot 121) = 2400 / 242 = 9.9174$
• $r = w_1 / w_0 = 2279 / 121 = 18.83$, so one positive counts as 18.83 negatives.
• Check: $2279 \cdot 0.5265 = 1200$ and $121 \cdot 9.9174 = 1200$. Each class carries half the total weight.

3. One SMOTE point. $x_i = (1.0, 2.0)$, neighbor $x_j = (1.5, 2.5)$, $\lambda = 0.4$. The difference is $(0.5, 0.5)$; times 0.4 gives $(0.2, 0.2)$; added to $x_i$ gives $(1.2, 2.2)$.

4. Metrics for the weighted model at t = 0.5. On test: $TP = 27$, $FP = 128$, $FN = 7$, $TN = 638$.

• Precision $= 27 / (27 + 128) = 27 / 155 = 0.174$
• Recall $= 27 / (27 + 7) = 27 / 34 = 0.794$
• F1 $= 2 \cdot 27 / (2 \cdot 27 + 128 + 7) = 54 / 189 = 0.286$
• MCC numerator $= 27 \cdot 638 - 128 \cdot 7 = 17226 - 896 = 16330$
• MCC denominator $= \sqrt{155 \cdot 34 \cdot 766 \cdot 645} = \sqrt{5270 \cdot 494070} = \sqrt{2603748900} = 51026.9$
• MCC $= 16330 / 51026.9 = 0.320$
• Accuracy $= (27 + 638) / 800 = 0.831$, lower than the useless model's 0.958, yet this model catches 79% of fraud.

5. Same model, different threshold. Tuned on validation, the best F1 threshold is 0.86 for the weighted model and 0.24 for the unweighted one. Convert 0.86 to the unweighted scale:

• Weighted odds $= 0.86 / 0.14 = 6.143$
• Divide by $r$: $6.143 / 18.83 = 0.3262$
• Back to a probability: $0.3262 / (1 + 0.3262) = 0.3262 / 1.3262 = 0.246$

That is the 0.24 found independently for the unweighted model. Both tuned models land at test recall 0.647 and F1 about 0.53, and their AUPRC is 0.548 versus 0.551. The weights moved the threshold; they did not make a better ranker.

6. Prior correction. A weighted model says $p_w = 0.6$. With $\pi = 121 / 2400 = 0.0504$, $r = 18.83$: odds $= 0.6 / 0.4 = 1.5$, divided by 18.83 gives $0.0797$, so $p = 0.0797 / 1.0797 = 0.0738$. A "60% fraud" score from the weighted model means about 7% fraud in reality.

4. Implementation

The full script is code/24-applied-ml/imbalanced_data.py. It uses only NumPy for the from-scratch parts and checks itself against scikit-learn.

Build it from scratch

Metrics. Count the four confusion cells once, then derive everything from them. The if ... else 0.0 guards handle the cases where nothing is predicted positive, which is common with rare classes.

def metrics(y, pred):
    tp, fp, fn, tn = confusion(y, pred)
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
    denom = np.sqrt(float((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn)))
    mcc = (tp * tn - fp * fn) / denom if denom else 0.0
    return {"precision": precision, "recall": recall, "f1": f1, "mcc": mcc}

Predict: for a model that flags every row as positive, which of precision, recall, F1, and MCC is 1, which is the positive rate, and which is 0?

Average precision. Sort by score, walk down the ranking, and record the precision each time you hit a positive. No threshold is involved; only the order matters.

def average_precision(y, scores):
    order = np.argsort(-scores)             # rank by score, highest first
    hits = y[order]
    precision_at_k = np.cumsum(hits) / np.arange(1, len(hits) + 1)
    return float(np.sum(precision_at_k * hits) / hits.sum())  # mean precision at each true positive

Predict: if you replace scores with 2 * scores + 1, does AP change?

Random over- and undersampling. Oversampling repeats positives until the classes match; undersampling keeps every positive and a random sample of negatives of the same size.

def random_oversample(X, y, rng):
    pos = np.where(y == 1)[0]
    extra = rng.choice(pos, size=int((y == 0).sum()) - len(pos), replace=True)  # duplicates
    keep = np.concatenate([np.arange(len(y)), extra])
    return X[keep], y[keep]

def random_undersample(X, y, rng):
    pos = np.where(y == 1)[0]
    neg = rng.choice(np.where(y == 0)[0], size=len(pos), replace=False)  # discards most negatives
    keep = np.concatenate([pos, neg])
    return X[keep], y[keep]

Predict: with 2,279 negatives and 121 positives, how many rows does each function return, and how many distinct positive rows does the oversampled set contain?

SMOTE. For each new point: pick a real minority point, pick one of its $k$ nearest minority neighbors, and place the new point at a random spot on the segment between them. Distances are computed among minority points only.

def smote(X_min, n_new, k, rng):
    d = np.linalg.norm(X_min[:, None, :] - X_min[None, :, :], axis=2)  # [m, m] pairwise distances
    np.fill_diagonal(d, np.inf)                                        # a point is not its own neighbor
    neighbors = np.argsort(d, axis=1)[:, :k]                           # [m, k] k nearest minority points
    base = rng.integers(0, len(X_min), size=n_new)                     # which real point to start from
    pick = neighbors[base, rng.integers(0, k, size=n_new)]             # one of its neighbors
    t = rng.random((n_new, 1))                                         # position along the segment
    return X_min[base] + t * (X_min[pick] - X_min[base])

Predict: a lone positive sits deep inside the negative cloud. Where do the synthetic points that start from it end up, and why is that a problem?

Weighted logistic regression. Plain gradient descent on cross-entropy, except each row's error is multiplied by its weight before it enters the gradient. Dividing by the total weight keeps the step size comparable to the unweighted version.

def fit_logreg(X, y, sample_weight=None, lr=0.5, epochs=2000):
    w_s = np.ones(len(y)) if sample_weight is None else sample_weight
    w, b = np.zeros(X.shape[1]), 0.0
    for _ in range(epochs):
        err = (sigmoid(X @ w + b) - y) * w_s           # each row's gradient scaled by its weight
        w -= lr * (X.T @ err) / w_s.sum()
        b -= lr * err.sum() / w_s.sum()
    return w, b

def balanced_weights(y):
    n, counts = len(y), np.bincount(y.astype(int))     # same rule as sklearn class_weight="balanced"
    per_class = n / (len(counts) * counts)
    return per_class[y.astype(int)], per_class

Predict: weighting every positive by 18.83 and oversampling every positive 18.83 times give the same loss in expectation. Which one is cheaper to train?

Threshold tuning. Sweep a grid of thresholds on the validation set and keep the one with the best F1. Swap F1 for expected cost or a recall floor when the business has one.

def best_threshold(y, probs):
    grid = np.round(np.arange(0.05, 0.96, 0.01), 2)
    f1s = np.array([metrics(y, (probs >= t).astype(int))["f1"] for t in grid])
    return float(grid[f1s.argmax()]), grid, f1s

Predict: the validation set has only 45 positives. If you rerun with a different seed, how much do you expect the chosen threshold to move?

Use the library

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, classification_report, matthews_corrcoef
from sklearn.model_selection import TunedThresholdClassifierCV   # scikit-learn >= 1.5
from imblearn.over_sampling import SMOTE                          # pip install imbalanced-learn
from imblearn.pipeline import Pipeline

# 1. Class weights: one argument.
weighted = LogisticRegression(class_weight="balanced").fit(X_train, y_train)

# 2. Threshold tuning with internal cross-validation (no hand-written loop, no test peeking).
tuned = TunedThresholdClassifierCV(LogisticRegression(), scoring="f1", cv=5).fit(X_train, y_train)
print(tuned.best_threshold_, average_precision_score(y_test, tuned.predict_proba(X_test)[:, 1]))

# 3. SMOTE inside a pipeline, so resampling only ever touches the training folds.
smote_model = Pipeline([("smote", SMOTE(k_neighbors=5, random_state=0)),
                        ("clf", LogisticRegression())]).fit(X_train, y_train)
print(classification_report(y_test, smote_model.predict(X_test), digits=3))

What the library adds: class_weight="balanced" computes exactly $n / (C \cdot n_c)$; TunedThresholdClassifierCV tunes the cut-off with cross-validation and refits; imbalanced-learn's Pipeline applies samplers during fit only, never during predict, which is what keeps SMOTE from leaking into evaluation. Tree boosters have their own knob: scale_pos_weight in XGBoost and LightGBM, typically set near $n_0 / n_1$.

Run it

From guide/:

uv run code/24-applied-ml/imbalanced_data.py

Key output:

positives: train 121/2400, val 45/800, test 34/800  (synthetic)
always-negative on test: accuracy 0.958, recall 0.000, F1 0.000, MCC 0.000
class weights: n=2400, n_pos=121, w_neg=0.5265, w_pos=9.9174, ratio=18.83
best F1 threshold for baseline (tuned on validation): 0.24
best F1 threshold for class weights (tuned on validation): 0.86

strategy                 thresh   prec  recall     F1    MCC  AUPRC
baseline                   0.50  0.652   0.441  0.526  0.520  0.548
oversample                 0.50  0.174   0.794  0.286  0.320  0.551
undersample                0.50  0.172   0.794  0.283  0.317  0.540
SMOTE                      0.50  0.175   0.794  0.287  0.321  0.552
class weights              0.50  0.174   0.794  0.286  0.320  0.551
baseline + tuned t         0.24  0.458   0.647  0.537  0.521  0.548
class weights + tuned t    0.86  0.449   0.647  0.530  0.515  0.551
AUPRC of a random scorer = positive rate = 0.043
checks passed: matches sklearn class_weight='balanced' on 100.0% of test rows

Read the AUPRC column: every strategy ranks positives almost equally well (0.540 to 0.552). The F1 column differs because each strategy sits at a different operating point. On this data, the four rebalancing methods at threshold 0.5 overshoot toward recall, and tuning the threshold beats all of them.

5. Engineering

Start with the scoreboard. Pick the metric from the decision the model drives. Use a cost-weighted metric if you can price a false negative and a false positive (classification metrics shows how). Use recall at a fixed precision (or precision at a fixed alert budget) when a team can only review so many alerts a day. Report AUPRC for ranking quality and always state the positive rate next to it, since that is its random baseline.

Default recipe. A stratified split, a strong model on the raw data, then tune the threshold on validation. Add class weights if the learner struggles to fit the minority class at all (common with gradient boosting on 1:1000 data, or with neural networks where most batches hold no positives). Reach for resampling when you need it for speed (undersampling huge negative sets) or when a specific model benefits; measure it rather than assuming.

Calibration. Resampling and class weights distort predicted probabilities toward 50%. That matters when probabilities are used as probabilities: risk scores, expected-cost decisions, pricing. Apply prior correction or recalibrate on untouched validation data (CalibratedClassifierCV). Research on clinical risk models (van den Goorbergh et al., JAMIA 2022) found imbalance corrections made logistic models badly miscalibrated without improving discrimination.

Undersampling at scale. With billions of negatives (ad clicks, search logs), downsampling negatives is standard because it cuts training cost. Keep the sampling rate and correct predictions afterwards with the same odds formula, or weight the kept negatives by the inverse sampling rate.

SMOTE limits. It interpolates in feature space, so it needs meaningful distances: scale features first, and it does not suit raw categorical data (use SMOTENC or SMOTEN), text, or images. It amplifies label noise and outliers, and it can paint synthetic positives into negative regions. Variants such as Borderline-SMOTE and ADASYN change which points seed new samples, not the core limitation.

Data size. With only a few dozen positives, every estimate is noisy: the tuned threshold, F1, even AUPRC. Use repeated stratified cross-validation, report intervals, and spend effort on getting more labeled positives, which usually beats any resampling trick.

Monitoring. The positive rate drifts (fraud waves, seasonal disease). Precision at a fixed threshold drifts with it. Track the live positive rate and alert volume, and revisit the threshold when either moves.

Common mistakes

• Resampling before the train/test split, or before cross-validation folds are made. Synthetic or duplicated positives leak into evaluation and inflate every metric.
• Evaluating on a rebalanced test set. The test set must keep the real class ratio, or precision is meaningless.
• Tuning the threshold on the test set, then reporting test performance as if it were unseen.
• Reporting accuracy or ROC AUC alone on a 1% positive problem, or AUPRC without the positive rate.
• Treating a class-weighted model's output as a calibrated probability.
• Applying SMOTE to one-hot categorical columns and producing "0.37 of a country".

6. Knowledge check

Practice exercise. A fraud model reviewed on 10,000 transactions (50 fraud) gives $TP = 40$, $FP = 160$, $FN = 10$. Compute accuracy, precision, recall, F1, and MCC. Then compute the balanced class weights for the training set this came from if it had the same ratio.

Implementation challenge. Add a cost-based threshold to the script: given $c_{FN} = 50$ and $c_{FP} = 1$, pick the validation threshold that minimizes $c_{FN} \cdot FN + c_{FP} \cdot FP$ for the baseline model, and report the test cost next to the F1-tuned threshold's cost. Then check it against the theoretical cut-off for a calibrated model, $t = c_{FP} / (c_{FP} + c_{FN})$.

def best_cost_threshold(y, probs, c_fn=50, c_fp=1):
    grid = np.round(np.arange(0.01, 0.96, 0.01), 2)
    costs = []
    for t in grid:
        tp, fp, fn, tn = confusion(y, (probs >= t).astype(int))
        costs.append(c_fn * fn + c_fp * fp)
    return float(grid[int(np.argmin(costs))])

Summary

• Accuracy is dominated by the majority class; use precision, recall, F1, MCC, cost, and AUPRC, and quote AUPRC with the positive rate.
• Oversampling, undersampling, SMOTE, and class weights all push the boundary toward the majority class; for a linear model they mainly change the operating point, not the ranking.
• Class weights of $n / (C \cdot n_c)$ multiply the predicted odds by about $n_0 / n_1$; divide by that ratio to recover real-world probabilities or to translate thresholds.
• Tune the threshold on validation (or with TunedThresholdClassifierCV) against the metric the business cares about. It is often the whole fix.
• Resample only inside training folds, keep the test set at the true ratio, and recalibrate if anyone reads the scores as probabilities.

Next: Hyperparameter tuning

Related: Classification metrics and thresholds · Logistic regression · Anomaly detection · Feature engineering

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Handling Imbalanced Data.

Interview angle