Loading slide

Why Shallow Networks Failed

  1. 01Why Shallow Networks Failed
  2. 02Memory card
  3. 03What "deep" actually means
  4. 04The vanishing gradient
  5. 05A small step inside each neuron
  6. 06Why bother bending the number?
  7. 07The bend they chose
  8. 08How the flat ends starve the signal
  9. 09The absurdly simple fix
  10. 10Take a breath
  11. 11Other fixes
  12. 12Depth vs. width
  13. 13Memory card
  14. 14The research gap
  15. 15Reinforce your understanding
  16. 16Question: The vanishing gradient
  17. 17Question: Why does a simple fix work?
  18. 18Question: Why does depth matter?
  19. 19Quiz: answer
  20. 20The one idea to keep
  21. 21The gradient problem was solved. What was still missing was scale.
  22. 22Want to go deeper?
14 / 21
BackNext

The research gap

Through the 1990s and into the 2000s, neural networks existed, but they stayed shallow. One or two hidden layers at most. Anything deeper was unreliable.

Other approaches dominated. Statistical methods and classical machine learning algorithms consistently beat neural networks on the benchmarks that mattered. Neural networks weren't the clear answer. They were a minority bet.

A small group of researchers kept working on them anyway. Geoffrey Hinton at Toronto, who you met back when backpropagation was first invented. at NYU. at Montreal. They published. They refined the techniques. They waited.

The problem wasn't that the ideas were wrong. The gradient fixes, ReLU, better initialization, normalization, were all within reach. The deeper issue was that none of it mattered much at the scale available then. The networks were too small. The datasets were too small. The computers were too slow.

The ideas needed the world to catch up. It was starting to.

Did you know?

Backpropagation, the method that makes all of this possible, was made practical for neural networks back in 1986, but the breakthrough that proved deep networks worked did not arrive until 2012. That is twenty-six years of a right idea waiting for the world, the data and the machines, to catch up. It is a common story. The laser was dismissed in 1960 as "a solution looking for a problem," and only later came to power barcode scanners, eye surgery, and the cables that carry the internet. The pieces are often invented long before anyone has a use for them (or can make use of them).
Citations(2)↓
  1. 1. proceedings.mlr.press
  2. 2. ibm.com