Skip to content
ML to LLMs
Labs / Classical ML

Classification threshold

Move the threshold and watch the confusion matrix, precision, recall, F1, and business cost change.

Theory and worked examples: Classification metrics and thresholds

Classification threshold lab

400 synthetic transactions, about 30% fraud. The model outputs a score; the threshold turns it into an action.

Score histogram by true class00.20.40.60.810204060model scorecountflag →
legitimatefraudthreshold
predicted fraudpredicted legitactually fraud
TP
58
caught
FN
66
missed · cost 660
actually legit
FP
24
false alarm · cost 24
TN
252
correctly ignored
ROC curve00.20.40.60.8100.20.40.60.81false positive ratetrue positive rate (recall)

Every threshold is one point on this curve. The dashed diagonal is random guessing.

Calibration (reliability) curve00.20.40.60.8100.20.40.60.81mean predicted scoreobserved fraud rate

Calibrated: a score of 0.3 means about 30% of such cases are fraud. Toggle the overconfident model to see miscalibration.

At threshold 0.50 the model flags 82 transactions: 58 real fraud and 24 false alarms, and misses 66 fraud cases. With a false negative costing 10 and a false positive 1, the cheapest threshold is 0.09. Missed fraud is the expensive error, so the best threshold sits low. Note that accuracy (77.5%) barely reflects any of this: always predicting "not fraud" would score 69%.