Logistic regression

1. Intuition

Linear regression outputs any real number, but a probability must lie in $[0, 1]$. Logistic regression keeps the linear score, called the logit $z = w \cdot x + b$, and passes it through the sigmoid $\sigma(z) = 1/(1 + e^{-z})$:

• very negative $z$ gives a probability near 0,
• $z = 0$ gives exactly 0.5,
• very positive $z$ gives a probability near 1.

The boundary where the model is undecided ($p = 0.5$) is where $z = 0$, a straight line (or flat plane) in feature space.

Training uses cross-entropy: the negative log of the probability the model gave the correct label. Predicting 0.9 for a true positive costs $-\ln 0.9 = 0.105$; predicting 0.01 costs $-\ln 0.01 = 4.6$. Confident mistakes are punished hard.

2. Visualization

Synthetic data. Probability changes smoothly across the boundary; the model is most uncertain near the black line.

3. The math

Symbols

Model and loss

Log-odds interpretation

Rearranging the sigmoid: $\ln\frac{\hat{p}}{1 - \hat{p}} = w \cdot x + b$. Increasing feature $j$ by one unit adds $w_j$ to the log-odds, which multiplies the odds by $e^{w_j}$.

Gradient

Using $\sigma'(z) = \sigma(z)(1 - \sigma(z))$ and the chain rule, the sigmoid's derivative cancels:

Same form as linear regression: (prediction − target) × input.

Multiclass: softmax and categorical cross-entropy

Worked example 1: one prediction

One feature (hours studied) with $w = 1.5$, $b = -4$. A student studied $x = 3$ hours.

1. Logit: $z = 1.5 \times 3 - 4 = 0.5$.
2. Probability: $\hat{p} = 1/(1 + e^{-0.5}) = 1/(1 + 0.6065) = 0.6225$.
3. If they passed ($y = 1$): $L = -\ln 0.6225 = 0.474$. If they failed ($y = 0$): $L = -\ln(1 - 0.6225) = -\ln 0.3775 = 0.974$.
4. Gradient for $y = 1$: $\partial L/\partial w = (0.6225 - 1)(3) = -1.133$, so a gradient step increases $w$.
5. Odds: $0.6225/0.3775 = 1.65$. Each extra hour multiplies the odds by $e^{1.5} = 4.48$.

Worked example 2: softmax

Logits $[2, 1, 0.1]$. Exponentials $[7.389, 2.718, 1.105]$, sum $11.212$. Probabilities $[0.659, 0.242, 0.099]$. If class 0 is correct, $L = -\ln 0.659 = 0.417$.

4. Implementation

You will build binary logistic regression and its multi-class form (softmax regression) from scratch in NumPy, with the two numerical guards every real implementation has, then confirm the probabilities match scikit-learn.

Build it from scratch

The sigmoid from section 3, written so it never overflows. np.exp(1000) is infinity, so the naive 1 / (1 + np.exp(-z)) warns for large negative z. Computing exp(-|z|) keeps the exponent at or below zero for every input.

def sigmoid(z):
    # Stable for large |z|: never calls exp on a large positive number
    e = np.exp(-np.abs(z))                                 # always in (0, 1]
    return np.where(z >= 0, 1 / (1 + e), e / (1 + e))

def bce(y, p, eps=1e-12):
    p = np.clip(p, eps, 1 - eps)                           # log(0) guard
    return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))

Predict: without the clip, what is the loss on one example where the model outputs p = 1.0 exactly but the label is 0?

Training is gradient descent on mean binary cross-entropy. The gradient is the tidy $(p - y)\,x$ from worked example 1, averaged over rows: the same shape as linear regression's, with a probability in place of a raw prediction.

def fit_logistic(X, y, lr=0.5, steps=3000):
    n, d = X.shape
    w, b = np.zeros(d), 0.0
    for _ in range(steps):
        p = sigmoid(X @ w + b)                              # [n] probabilities
        w -= lr * X.T @ (p - y) / n                         # gradient of mean BCE: (p - y) x
        b -= lr * np.mean(p - y)
    return w, b

Predict: if the training data is perfectly separable (some line puts every 0 on one side and every 1 on the other), what happens to w as you keep training?

Softmax turns k logits into k probabilities. Subtracting each row's maximum first changes nothing mathematically (it cancels in the ratio) but keeps exp from overflowing.

def softmax(Z):
    Z = Z - Z.max(axis=1, keepdims=True)                   # subtract row max: same result, no overflow
    E = np.exp(Z)
    return E / E.sum(axis=1, keepdims=True)

Softmax regression has one weight column per class, W of shape [d, k]. With one-hot labels Y, the gradient has exactly the binary form, (P − Y) times the inputs.

def fit_softmax(X, y, k, lr=0.5, steps=3000):
    n, d = X.shape
    W, b = np.zeros((d, k)), np.zeros(k)
    Y = np.eye(k)[y]                                        # one-hot, [n, k]
    for _ in range(steps):
        P = softmax(X @ W + b)                              # [n, k]
        W -= lr * X.T @ (P - Y) / n                         # same (p - y) x shape as the binary case
        b -= lr * (P - Y).mean(axis=0)
    return W, b

Predict: softmax([[1000, 1001, 1002]]) with and without the max subtraction. What does each return?

Use the library

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

clf = make_pipeline(StandardScaler(), LogisticRegression(C=1.0, max_iter=1000))   # C = 1/λ (L2)
clf.fit(X_train, y_train)
proba = clf.predict_proba(X_test)[:, 1]   # probabilities; choose the threshold separately

What the library adds: L2 regularization on by default (C is the inverse strength, so smaller C means stronger shrinkage, the fix for the separable case), a second-order solver (L-BFGS) that needs no learning rate, and automatic softmax when y has more than two classes. To compare with the scratch code, the script turns the penalty off with C=np.inf.

Run it

From guide/:

uv run code/04-classification/logistic_scratch.py

sigmoid(-1000), sigmoid(0), sigmoid(1000) = [0.  0.5 1. ]
binary: BCE 0.4567, accuracy 0.824, w = [ 2.636 -0.871  0.114  1.285]
sklearn (no penalty): w = [ 2.636 -0.871  0.114  1.285]
softmax: rows sum to 1: True, accuracy 0.963
sklearn multinomial accuracy 0.958
checks pass: scratch probabilities match scikit-learn

Both datasets are synthetic (make_classification, make_blobs). The worked examples, the decision-boundary figure, and the cross-entropy figure come from code/04-classification/classification.py. Full scratch script: code/04-classification/logistic_scratch.py.

5. Engineering

Good use cases. Baselines for any classification problem; sparse high-dimensional data such as TF-IDF text; credit and risk models that must be explained; any setting where calibrated probabilities matter.

Regularization. scikit-learn applies L2 by default with strength C = 1/λ (smaller C means more regularization). Without regularization, perfectly separable data drives weights to infinity.

Preprocessing. Scale features (for the optimizer and the penalty); one-hot categoricals; add interaction or polynomial features if the boundary should curve.

Cost. Training is fast (convex, solved with L-BFGS or similar); inference is one dot product.

Thresholds are separate. predict uses 0.5, which is rarely the right business threshold. Use predict_proba and choose the threshold from error costs (next lesson).

Watch out

Failure modes. Nonlinear boundaries (XOR-like patterns) that a linear score can't separate; perfect separation without regularization; correlated features making weights unstable and hard to interpret; class imbalance making the 0.5 threshold useless.

Common mistakes

• Using MSE instead of cross-entropy (the loss becomes non-convex in $w$ and gradients vanish).
• Computing log(sigmoid(z)) directly for large negative $z$; use stable library losses.
• Reading coefficients without scaling features first.
• Treating predicted probabilities as calibrated without checking.

6. Knowledge check

Practice exercise. With $w = [2, -1]$, $b = 0.5$, and $x = [1, 3]$, compute $z$, $\hat{p}$, and the loss if $y = 0$.

Implementation challenge. Train fit_logistic on a synthetic 2D dataset, check your weights against scikit-learn with penalty=None, and plot the decision boundary $w_1x_1 + w_2x_2 + b = 0$.

Summary

• Logit $z = w\cdot x + b$; probability $\hat{p} = \sigma(z)$; the boundary $z = 0$ is linear.
• Binary cross-entropy is $-\ln$ of the probability of the true label; confident mistakes cost a lot.
• The gradient is $(\hat{p} - y)x$, the same shape as linear regression.
• Each weight changes the log-odds; $e^{w_j}$ is the odds multiplier.
• Softmax plus categorical cross-entropy generalizes to many classes and is the output layer of neural classifiers and LLMs.

Next: Classification metrics and thresholds

Related: Linear regression · Forward pass · Model card: logistic regression

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Logistic Regression.

Interview angle