The walk does not care how far

One question is still open. The walk crossed a single hidden layer. Real networks have many, and a weight in the first layer of a hundred-layer network is a very long way from the price.

So put another layer in and see what breaks.

Give this machine two hidden rows instead of one, as in the picture.

The first row is the one you already know: four units, one per wine type. The second row is new. It has two units, and it groups the wine types by whether the pairing works. Plummy cabernet and apricotty viognier are the two that suit each other, so they feed the matched unit. The other two feed mismatched. The price is then read off those two signals.

A weight at the front now sits four steps from the answer instead of three.

Watch what that changes for the backward walk. The error arrives at the price. It crosses the last weights and becomes a fact about the pairing units. It crosses one more set of weights and becomes a fact about the wine-type units. It crosses one more and lands on the tastes.

Nothing was added to the method. The same sentence from slide 26 still describes every crossing: arrive with a rate, multiply by the local thing, leave with a new rate for whatever sits one step further back. A second layer just means the sentence runs one more time.

That is the property worth having. The walk is indifferent to how far it has to travel, because it never travels far. It only ever takes one step, and one step looks the same wherever it happens.

So the answer for ten layers is the same, and for a hundred, and for a thousand. The route grows. The rule does not.

There is an honest catch, and it is not a small one. Each crossing multiplies the running number by something. Cross a hundred of them and those multiplications compound, the way repeated interest does. If the numbers along the way are mostly below 1, the rate arriving at the front layers can shrink to almost nothing, and those weights barely move. If they are mostly above 1, it can swell until the steps are wildly too big.

The rule still applies at every step. Getting a deep network to actually learn is a separate craft, and much of the last few decades of the field has gone into it.

The walk does not care how far

One question is still open. The walk crossed a single hidden layer. Real networks have many, and a weight in the first layer of a hundred-layer network is a very long way from the price.

So put another layer in and see what breaks.

Give this machine two hidden rows instead of one, as in the picture.

A weight at the front now sits four steps from the answer instead of three.

That is the property worth having. The walk is indifferent to how far it has to travel, because it never travels far. It only ever takes one step, and one step looks the same wherever it happens.

So the answer for ten layers is the same, and for a hundred, and for a thousand. The route grows. The rule does not.

The rule still applies at every step. Getting a deep network to actually learn is a separate craft, and much of the last few decades of the field has gone into it.