Watch the correction fade

Think of compound interest running backwards.

Money in a savings account grows because each year multiplies what the last year left. Small percentages become large sums, given enough years. A correction travelling back through a network is also multiplied repeatedly. In this example, every factor is smaller than one, so the correction compounds down.

Suppose the correction starts at 100 and each step passes back a fifth of what it received.

100 → 20 → 4 → 0.8

The exact factors vary across networks and training steps. They are not always smaller than one. A product of many small factors can vanish, while a product of large factors can explode. What matters here is that repeated multiplication can change the signal rapidly. Keep 90% at every layer and after thirty layers only 4 percent of the original value is left.

This is called a vanishing gradient. The gradient measures how the error responds to small changes in the weights. Vanishing means that this sensitivity has become almost zero before it reaches the first layers, so their weights receive little guidance.