Loading slide
Backpropagation has now handed every weight a number: how much the final error would change if that weight moved a little.
Knowing that is not the same as doing anything about it. Nothing has moved yet.
Which is where Chapter 1 comes back, because the rule that moves a weight is one you have already used:
weight change = error x unit x learning rate
That is the delta rule from the first wine machine. The error said how big the repair should be and in which direction. The unit said how much that particular weight had to do with causing it. The learning rate, 0.1 in that chapter, kept the whole thing down to a nudge.
Nothing about that changes here. In Chapter 1 a weight's effect was easy to read off, because every weight touched the price directly, so its effect was simply the input it carried. That is what unit meant in the rule.
For a weight buried behind a hidden unit, that shortcut is gone, and backpropagation supplies the effect instead. The cabernet weight entering hidden 1 has an effect of 25, worked out on the last slide by crossing the output weight. So its change is error x 25 x learning rate.
Same rule, same three ingredients. The only difference is where the middle number comes from.
It is worth seeing how small the resulting move is. With a learning rate of 0.001, the cabernet weight changes by 16.25 x 25 x 0.001, which is about 0.41. It goes from .20 to roughly .61, not to 1.20.
The last slide imagined moving it by a whole 1 only to make the measurement readable. Training never takes a step that size.
Move the weights by those small amounts and the guess climbs from $23.75 to somewhere near $34. The bottle costs $40, so the machine is still short, on purpose. This is the Chapter 1 lesson again: close the gap partway, let the next bottle have its say, and never let one wine seize control of the weights.
This has a name, and it is worth having, because it is one of the most common phrases in the field.
Moving every weight a small step in the direction that lowers the error is called gradient descent.
Both halves of that name are ordinary words doing ordinary jobs.
Descent means going down. The thing being descended is the error. The machine is $16.25 wrong, and each step is meant to leave it slightly less wrong. Picture walking downhill in fog. You cannot see the valley, but you can feel which way the ground falls away under your feet, and you can take a step that way. Then you feel again.
Gradient is the word for that felt slope, collected across every weight at once. The backward walk found, for each weight, whether the error rises or falls when the weight moves and by how steeply. Gather all of those together and you have the gradient. It is not a new quantity. It is the whole set of effects you just calculated, under one word.
The fog matters in the picture. The machine has no map of the error and no idea where the lowest point is. It knows only the slope directly under the weights it currently has, and it can only take one small step before it has to feel around again. That is why training is a loop rather than a solution.
The learning rate is the size of that step, and Chapter 1 already showed what it costs to get wrong. Too small and you shuffle downhill for a thousand bottles. Too large and you stride clean over the valley and land higher up the far side, which is the shower tap running from freezing to scalding and back.
So the two names name two jobs, and it is worth keeping them apart.
Backpropagation works out the slope. It walks backwards from the error and gives every weight its effect. It moves nothing.
Gradient descent takes the step. It uses those effects, applies the learning rate, and changes the weights.
One measures. The other moves.
Loading contents...