Loading slide
Before we look at what AlexNet learned, it is worth slowing down on how it learned anything at all. Because nobody sat down and taught it. There was no rulebook, no human pointing out what a dog looks like. So where did the knowledge come from?
It came from one simple loop, run an astonishing number of times.
At the very start, the network knows nothing. All 60 million of its numbers are random. This is the same starting point as the word-guessing model from the last module, the one where "stars" began buried in the noise, no more likely than "banana." Show this network a photo of a dog and it does the same thing: answers with noise, a confident guess of "toaster," or "volcano," or anything at all. It is wrong, completely, and that is the point.
Here is the loop, the same one from the last module. Show the network one image. Let it guess. Compare its guess to the human label that came with the photo, the right answer. Measure how far off it was, as a single number: the error. Then send that error backward through every layer and nudge each of the 60 million numbers a tiny amount, just enough to make that one guess a little less wrong next time.
One image. One nudge. Then the next image, and the next nudge. And again.
Now sit with the scale of it. ImageNet held 1.2 million labeled photos. AlexNet went through all of them, then went through them all again, dozens of times over. That is tens of millions of guesses, each one followed by a tiny correction to those 60 million dials. No single nudge teaches the network anything you could point to. But stacked up, millions upon millions of them, they slowly carve the random starting numbers into a setting that recognizes images.
The knowledge was never poured in. It was worn in, one tiny correction at a time. Nobody wrote down what a dog looks like. The network was simply wrong, a few million times, and corrected a little after each one, until being right was all that was left.