The research gap

Through the 1990s and into the 2000s, neural networks existed — but they stayed shallow. One or two hidden layers at most. Anything deeper was unreliable.

Other approaches dominated. Statistical methods and classical machine learning algorithms consistently beat neural networks on the benchmarks that mattered. Neural networks weren't the clear answer. They were a minority bet.

A small group of researchers kept working on them anyway. Geoffrey Hinton at Toronto. Yann LeCun at NYU. Yoshua Bengio at Montreal. They published. They refined the techniques. They waited.

The problem wasn't that the ideas were wrong. The gradient fixes — ReLU, better initialization, normalization — were knowable. The deeper issue was that none of it mattered at the scale available then. Networks were too small. Datasets were too small. Computers were too slow.

The ideas needed the world to catch up. It was starting to.