Loading slide
The chip that could do that was the
Remember the orchestra from the very first module? We met two kinds of chip there. The CPU was the conductor: one brilliant mind, but only one beat at a time. The GPU was the orchestra: a hundred musicians, none of them more capable than the conductor, but all playing at once, filling the hall in an instant. Back then it was a way to draw graphics. Here it becomes the key to training.
It is worth remembering why the GPU was built like that. When a computer draws a 3D scene, it has to work out the color of millions of pixels, and the color of one pixel does not depend on its neighbor, so they can all be computed at the same time. That is what the chip was designed for: thousands of small workers, each slower on its own, all running side by side. The orchestra, in silicon.
Picture handing that job to the CPU instead, the lone conductor. It would have to color the millions of pixels one after another, then start the whole screen again for the very next frame. By the time it finished a single image, the moment to show it would be long gone. A fast-moving game would lurch along like a slideshow. The work is not hard, there is just far too much of it to do in a line.
If you have ever played games, you already know this in your gut. A good graphics card and a cheap one run the same game very differently. The good one is simply a bigger orchestra: it packs in more of those small workers, and feeds them faster, so it can color more pixels every instant. That is what buys you smoother motion and richer scenes. A weaker card has fewer hands on the job, so it either slows down or drops detail to keep up. The difference you feel is, almost entirely, how many things the card can do at the same time.
Training turned out to need precisely this. Think about a single layer of the network. It might have thousands of neurons, and every one of them is doing the same humble job at the same moment: take its inputs, multiply each by a weight, add them up. No neuron in that layer needs to wait for the neuron next to it. Just like the pixels on the screen, their sums do not depend on each other, so all thousands can be worked out at the same instant.
The layers themselves still go in order, the second cannot start until the first is done. But the flood of little multiplications inside each layer is independent, and that is exactly the kind of work a GPU's thousands of small workers were born to do all at once.
Did you know?