The GPU Moment

Sometimes an idea is waiting for the right machine.

The last chapter ended on a frustration. The fixes for deep networks existed, but training them on anything interesting was still too slow. The technique was ready. What was missing was scale. This chapter is where that scale finally arrived, from a direction nobody in AI was looking.

Training a large neural network is, at its heart, a problem of arithmetic. Enormous amounts of it, repeated billions of times. For years that arithmetic ran on hardware never built for it: processors designed to do one thing at a time, very fast, when what was actually needed was the opposite, the ability to do millions of small things all at once.

The machine that could do that already existed. It had been built for a completely different purpose, and it was sitting in plain sight.

This is a pattern worth noticing: breakthroughs in AI often come not from new ideas about intelligence, but from unexpected infrastructure arriving at the right moment. The math was ready. The concept was ready. The hardware was what had been missing.