Gradient descent

That repeated adjustment process is called .

Backpropagation calculates the gradient. Gradient descent uses it to update the weights in the opposite direction. Then the network tries again.

The name describes a picture. Imagine walking down a hill in thick fog. You cannot see the bottom, but you can feel which way the ground slopes under your feet and step that way.

Backpropagation measures the slope. Gradient descent takes a step downhill.

One setting provides the base size of the step. It is called the . Some optimizers adjust the effective step for different parameters.

Small steps can make training slow. They do not guarantee arrival at the best settings. Training can stall on a flat area or settle in a place that is better than the start but not the best available.

Every step means running examples through the network and calculating gradients for its trainable weights. More steps require more computation.

A step that is too large has the opposite problem. It can cross the bottom in one move and land higher up the opposite side, leaving the error worse than before.

The useful step is large enough to make progress and small enough not to leap past the bottom.