Put the network inside the loop

Reinforcement learning was already useful before deep networks entered the loop.

In 1988, Richard Sutton described temporal-difference learning. A learner did not always have to wait for the final result. A later estimate could correct an earlier estimate as events unfolded.

In the 1990s, Gerald Tesauro combined that idea with a neural network in TD-Gammon. The system became a strong backgammon player through play and feedback. An older learning rule and a trainable network had started to meet.

The deep networks from the previous chapters made that combination useful for much larger inputs.

A deep network can learn patterns shared by many images. Instead of memorising one exact picture, it can recognise something useful across different pictures, such as an opening on the left.

Put that network inside the consequence loop. An image goes in. The network judges the available actions, and the learner chooses one.

Then the world responds. The choice leads somewhere good or bad. That consequence becomes a correction, and the correction changes the network's weights.

Nobody supplies the correct action in advance. The learner discovers which choices work by trying them, then carries that lesson into similar situations.

The loop is still the same: act, see what happens, adjust. The network makes it possible to reuse what was learned when the next image is similar but not identical.

Putting deep networks inside the consequence loop is called deep reinforcement learning.

The next slide shows a real system built this way.