Calculus for ML

1. Intuition

Imagine driving: your position changes over time, and your speed is how fast it changes right now. Speed is the derivative of position.

For a function like $f(t) = t^2 - 4t + 5$, the derivative at a point is the slope of the tangent line there. A positive slope means increasing $t$ increases $f$; negative means it decreases $f$; zero means you are at a flat spot, possibly a minimum.

With many inputs, a partial derivative measures sensitivity to one input while holding the others fixed. The gradient collects all the partial derivatives into a vector. It points in the direction of steepest increase, so its negative points downhill: exactly where gradient descent steps.

The chain rule handles functions inside functions. If a change in $w$ changes $z$, and a change in $z$ changes the loss, the effect of $w$ on the loss is the product of the two sensitivities.

2. Visualization

What to notice:

• The slope is zero exactly at the minimum, $t = 2$.
• On the right, arrows are longer where the surface is steeper and always cross contour lines at right angles. The $y$ direction is steeper (coefficient 2), so arrows lean vertically.

3. The math

Symbols

Rules you will use constantly

Chain rule. If $L = g(z)$ and $z = h(w)$, then

Gradient. For $L(w_1, \dots, w_p)$, $\nabla L = \left[\frac{\partial L}{\partial w_1}, \dots, \frac{\partial L}{\partial w_p}\right]$.

Worked example 1: a derivative

$f(t) = t^2 - 4t + 5$, so $f'(t) = 2t - 4$. At $t = 3$: $f(3) = 9 - 12 + 5 = 2$ and $f'(3) = 2$. Setting $f'(t) = 0$ gives $t = 2$, the minimum, with $f(2) = 1$.

Numerical check: $\frac{f(3.001) - f(2.999)}{0.002} = \frac{2.002001 - 1.998001}{0.002} = 2.000$. Always use this trick to test hand-derived gradients.

Worked example 2: partial derivatives and a gradient

$L(x, y) = x^2 + 2y^2$. Treat $y$ as a constant to get $\partial L/\partial x = 2x$; treat $x$ as a constant to get $\partial L/\partial y = 4y$. At $(1, 1)$: $\nabla L = (2, 4)$. The negative gradient $(-2, -4)$ points toward the origin, steeper in $y$.

Worked example 3: the chain rule on squared error

For one example, $L = (y - \hat{y})^2$ with $\hat{y} = wx + b$. Let $r = y - \hat{y}$.

With $x = 2$, $y = 7$, $w = 1$, $b = 0$: $\hat{y} = 2$, so $\partial L/\partial w = 2(2 - 7)(2) = -20$. A step with learning rate 0.1 gives $w = 1 - 0.1(-20) = 3$, and the new prediction is 6, much closer to 7.

4. Implementation

You will compute derivatives numerically, use them to check an analytic gradient (the "gradient check" every framework author runs), and descend a loss surface. This is the core loop that gradient descent and backpropagation scale up.

Build it from scratch

A derivative is a slope, so measure it: nudge the input a tiny step each way and divide the change in output by the change in input. The central difference (x+h and x−h) is far more accurate than the one-sided version because the first-order errors cancel.

def derivative(f, x, h=1e-5):
    """Central difference: slope of the chord from x-h to x+h."""
    return (f(x + h) - f(x - h)) / (2 * h)

Predict: if you shrink h to 1e-15, does the estimate get better or worse? (Think about floating-point rounding when you subtract two nearly equal numbers.)

A gradient is one partial derivative per input. Nudge one coordinate at a time, holding the others fixed, exactly as in worked example 2.

def gradient(f, w, h=1e-5):
    """One central difference per coordinate: a vector of partial derivatives."""
    w = np.asarray(w, dtype=float)
    g = np.zeros_like(w)
    for i in range(w.size):
        e = np.zeros_like(w)
        e[i] = h
        g[i] = (f(w + e) - f(w - e)) / (2 * h)
    return g

Predict: a model with 1 million parameters. How many evaluations of the loss does this function need for one gradient? Why does that rule it out for training but not for testing?

Compare the numerical gradient to the analytic one you derived by hand. The relative error should be tiny (below about 1e-7); a value near 1e-2 means your formula has a bug.

def L(p):
    return p[0] ** 2 + 2 * p[1] ** 2

def grad_L(p):
    return np.array([2 * p[0], 4 * p[1]])            # analytic gradient

Gradient descent repeats one move: step against the gradient, scaled by the learning rate.

def descend(grad, w0, lr=0.1, steps=50):
    w = np.asarray(w0, dtype=float)
    for _ in range(steps):
        w = w - lr * grad(w)                         # step against the gradient
    return w

Predict: on $L = x^2 + 2y^2$, each step multiplies $y$ by $(1 - 4\cdot\text{lr})$. What happens to $y$ with lr = 0.6?

Use the library

Autograd frameworks compute exact gradients (up to float rounding), not approximations, by applying the chain rule through every operation you ran:

import torch

p = torch.tensor([1.0, 1.0], requires_grad=True)
(p[0] ** 2 + 2 * p[1] ** 2).backward()
print(p.grad)                     # tensor([2., 4.])

PyTorch records each operation in a graph during the forward pass, then walks it backward multiplying local derivatives. That is worked example 3 automated. torch.autograd.gradcheck runs the same numerical-versus-analytic comparison you just built.

Run it

From guide/:

uv run code/00-foundations/derivatives_scratch.py

f'(3): numerical 2.000000, analytic 2
∇L(1, 1): numerical [2. 4.], analytic [2. 4.], relative error 1.5e-12
chain rule dL/dw at w=1: -20.0, numerical -20.000000
one step, lr=0.1: w = 3.0, new prediction 6.0
gradient descent lr=0.1: from (1, 1) to [0. 0.]
gradient descent lr=0.6: from (1, 1) to [       0.     20248916.2398]
all checks pass

The chain-rule line reproduces worked example 3. Full script: code/00-foundations/derivatives_scratch.py. Figures: code/00-foundations/foundations.py.

5. Engineering

Gradient checking. Before trusting a hand-written gradient, compare it to finite differences on a few random inputs; relative error should be around $10^{-6}$ or smaller. Frameworks use automatic differentiation, which is exact, so you mostly need this when writing custom operations.

Numerical issues. $\ln(0)$ is $-\infty$ and $e^{1000}$ overflows. Libraries combine operations (log-softmax, BCE-with-logits) to stay stable; prefer them.

Where gradients vanish. The sigmoid's derivative is at most 0.25. Multiplying many such factors through the chain rule shrinks gradients toward zero in deep networks: the vanishing-gradient problem, covered in backpropagation.

Watch out

Failure modes. Sign errors (stepping uphill), forgetting the inner derivative in the chain rule, and using a finite-difference step $h$ so small that floating-point rounding dominates.

Common mistakes

• Treating a partial derivative as if other variables also change.
• Dropping the factor 2 from a squared term (it only rescales the gradient, but it changes the effective learning rate).
• Confusing $\log_{10}$ with $\ln$; ML formulas use natural logs unless stated.

6. Knowledge check

Practice exercise. For $L(w) = \ln(1 + e^{w})$, compute $dL/dw$ and evaluate it at $w = 0$.

Implementation challenge. Use numerical_grad to verify the analytic gradient of the linear-regression MSE ($\frac{2}{n}\sum(\hat{y}-y)x$) on random data.

Summary

• A derivative is a local slope; zero slope marks flat spots such as minima.
• Partial derivatives hold other inputs fixed; the gradient collects them and points uphill.
• The chain rule multiplies sensitivities along a chain of functions; backpropagation is this rule applied systematically.
• Check gradients with central finite differences.

Next: Probability and statistics

Related: Gradient descent · Backpropagation

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Calculus for Machine Learning.

Interview angle