Generalization

A student who memorizes every practice exam question will fail the real test if the questions change slightly. What you want is a student who actually understood the material: one who can handle questions they've never seen before.

Networks face exactly the same challenge. The ability to handle new examples is called generalization, and it is the central aim of training.

The way you test for it is straightforward.

Say you're training a network to recognize cats and dogs, and you have 10,000 labeled photos. Before training begins, you set 2,000 of them aside and hide them. The network trains on the other 8,000.

Once training is done, you show it those 2,000 photos for the very first time. If it gets them right, it generalized. If it only does well on the 8,000 it trained on, it memorized.

By the 1990s, researchers understood all of this. The theory was solid. So why didn't it take off?

Generalization

Networks face exactly the same challenge. The ability to handle new examples is called generalization, and it is the central aim of training.

The way you test for it is straightforward.

Say you're training a network to recognize cats and dogs, and you have 10,000 labeled photos. Before training begins, you set 2,000 of them aside and hide them. The network trains on the other 8,000.

Once training is done, you show it those 2,000 photos for the very first time. If it gets them right, it generalized. If it only does well on the 8,000 it trained on, it memorized.

By the 1990s, researchers understood all of this. The theory was solid. So why didn't it take off?