Loading slide

Why Shallow Networks Failed

  1. 01Why Shallow Networks Failed
  2. 02Memory card
  3. 03What "deep" actually means
  4. 04The vanishing gradient
  5. 05A small step inside each neuron
  6. 06Why bother bending the number?
  7. 07The bend they chose
  8. 08How the flat ends starve the signal
  9. 09The absurdly simple fix
  10. 10Take a breath
  11. 11Other fixes
  12. 12Depth vs. width
  13. 13Memory card
  14. 14The research gap
  15. 15Reinforce your understanding
  16. 16Question: The vanishing gradient
  17. 17Question: Why does a simple fix work?
  18. 18Question: Why does depth matter?
  19. 19Quiz: answer
  20. 20The one idea to keep
  21. 21The gradient problem was solved. What was still missing was scale.
  22. 22Want to go deeper?
8 / 21
BackNext

How the flat ends starve the signal

Remember what the backwards signal is actually asking at each step: "if I nudged this neuron a little, how much would the answer change?" That is the whole job of the training signal. It measures how much a small change here matters, so the network knows how hard to adjust each weight.

Now look at one of those flat ends of the S. Where the curve is nearly flat, nudging the input barely moves the output. A big push in, a tiny wiggle out. So the answer to "how much would a nudge matter?" comes back as "almost nothing." And a sigmoid spends most of its range out on those flat ends.

That "almost nothing" is a small fraction, often around 0.2. So every time the backwards signal crosses a sigmoid, it gets multiplied by roughly 0.2. Multiply by a fraction and a number shrinks. Then it reaches the next layer and shrinks again. And the next.

Send the correction back through the network and watch where it goes. It leaves the output at full strength, but keeping only a fifth, then a fifth of that fifth, then a fifth again, by the time it reaches L1 at the front there is almost nothing left. The signal does not fade gently. It falls off a cliff.

That is the vanishing gradient, in plain arithmetic: a bend whose flat ends multiply the signal down, layer after layer after layer, until the front of the network is learning from a whisper of a whisper.

Citations(2)↓
  1. 1. deeplearningbook.org
  2. 2. ibm.com
Citations(2)↓
  1. 1. deeplearningbook.org
  2. 2. ibm.com