How gradient descent finds the bottom

Watch an optimizer walk downhill on a loss surface, see why a learning rate that's too big makes it zigzag and then explode, and how momentum and Adam fix the zigzag.

6 min · Scroll, and the lab follows along. You can still drag every control. Full lesson: Gradient descent.

Gradient descent lab

Narrow bowl. Click the surface to choose a starting point.

Gradient descent−η∇L (next step)darker = higher loss; ring = minimum

Training a model means finding the parameters with the lowest loss. With two parameters you can draw that search as a map. This explainer walks through it one picture at a time; on the website the lab beside the text changes as you scroll.

The map

Each point on this surface is one choice of the two parameters, $\theta_1$ and $\theta_2$ . The colour is the loss at that point: darker blue is worse. The ring marks the lowest point, the minimum.

We start at the dot. We can't see the whole map, only the slope under our feet, so we need a rule that uses just that slope.

Which way is downhill?

The gradient $\nabla L$ points uphill, in the direction where the loss grows fastest. So we go the other way. The teal arrow is the step we're about to take:

\theta \leftarrow \theta - \eta \, \nabla L(\theta)

$\eta$ , the learning rate, scales how far we move. A steep slope gives a long arrow, a gentle slope a short one.

Step, recompute, repeat

Take the step, compute the new slope, take another. Nothing tells the optimizer to slow down near the bottom. It happens on its own, because the gradient shrinks as the surface flattens, so the steps shrink with it.

This bowl is narrow: steep across, shallow along. The path first drops across the steep direction, then creeps along the valley floor. That slow crawl is where most of the training time goes.

Too big a step

Raise $\eta$ from 0.10 to 0.18 and the path zigzags. In the steep direction each step overshoots the valley floor and lands on the opposite wall, which slopes back the other way.

It still gets there, because each overshoot is a little smaller than the one before. But it wastes steps bouncing between the walls.

Much too big

At $\eta = 0.21$ the bounce grows instead of shrinking, and the loss explodes. On this surface the steep direction has curvature 10, and gradient descent is only stable while

\eta \times 10 < 2 \quad\Longrightarrow\quad \eta < 0.2

This is why a loss that suddenly jumps to NaN usually means the learning rate is too high, not that the data is broken.

Smarter steps

Same surface, three optimizers, a small $\eta = 0.03$ and 60 steps each. Plain gradient descent (blue) still crawls along the valley floor. Momentum (orange) keeps a running velocity, so it keeps speeding up while the gradient points the same way. It builds up so much speed that it overshoots and swirls around the minimum, yet it still ends up closest. Adam (purple) scales each direction by how large its gradients have been, so the steep and shallow directions get similar-sized steps and it heads straight for the minimum. Each of its steps is only about $\eta$ long, though, so at this small $\eta$ it is still on its way.

That's why most deep networks today are trained with Adam or a close relative.

Try it yourself: click anywhere on the surface to move the starting point, or open the full lesson for the math with every step worked out.