What did we leave out?

Remember where training ended. The four output weights began at 25. After training, they became 40, 10, 10, and 40.

That worked because we had already built four perfect wine-type signals. Each hidden unit had a meaning we chose, and it reported exactly 1 or 0.

Now look at the wires entering the middle row. Can you see what is missing?

Press what is missing.

Yes. Those wires had no adjustable weights.

That was our simplification. We chose the two tastes each middle unit listened to. In effect, the chosen wires counted and the other wires did not exist. We also built each middle unit to answer only 1 or 0.

The neat names, such as plummy cabernet, came from us too. The middle row did not learn any of that.

A learning network cannot begin knowing which tastes belong together. Each middle unit needs a possible connection from every taste, with an adjustable weight on every connection.

When those weights appear, the neat wine-type names disappear. The units begin as hidden 1, hidden 2, and so on. Training must make their signals useful.

The old output weights no longer sit beside reliable wine-type signals, so they cannot be treated as finished either. In the revealed network, all four output weights return to the starting value 25.

The incoming and outgoing weights will now learn together.