Loading slide
The speedup was expected. What wasn't expected was what happened to the models themselves.
Researchers assumed bigger networks would be better the way a bigger engine is better: more of the same, proportionally stronger. A network twice the size should be roughly twice as good. That is not what they found.
Past a certain scale, something changed in kind, not just in degree. Networks began noticing things nobody had told them to look for. Patterns appeared in the hidden layers that no one had designed or anticipated. Tasks that used to demand painstaking hand-engineering started solving themselves, simply from exposure to enough data at enough size.
We have a name for this already. Back with the neurons, we called it emergence: many simple parts, put together in great enough number, start behaving as something more than their sum. One water molecule is not wet, yet trillions of them together are. The piece didn't change. The scale did.
Here it was happening one level up. Not single neurons adding up to a judgment, but whole networks crossing a size where new behavior appeared, behavior nobody had built in. The training method hadn't gotten smarter. It was the same backpropagation from 1986, the one Hinton and his colleagues published. What changed was scale: more data to learn from, more weights to learn with, more compute to run it all. Turning up that dial didn't just make the results a bit better. It made them qualitatively different.
Scale itself was the missing piece. Bigger networks did not just improve. Past a point, they behaved differently.