Loading slide

A Network That Learns

  1. 01A Network That Learns
  2. 02Stacking neurons
  3. 03Quick recall
  4. 04What stacking does
  5. 05Hidden layers
  6. 06The problem of credit
  7. 07Backpropagation
  8. 08Sharing the blame
  9. 09The search problem
  10. 10Fog on a hill
  11. 11Gradient descent
  12. 12Just to be clear
  13. 13What the hidden layer learns
  14. 14What the hidden layer discovers
  15. 15Did it actually learn?
  16. 16Generalization
  17. 17Limits in the 1990s
  18. 18Reinforce your understanding
  19. 19Question: Backpropagation
  20. 20Question: What the hidden layer learns
  21. 21Question: Memorizing vs. generalizing
  22. 22Quiz: answer
  23. 23The method existed. What was missing was scale.
  24. 24Want to go deeper?
2 / 23
BackNext

Stacking neurons

You've seen what a single neuron does: it takes in inputs, multiplies each one by a weight, adds them up, and fires if the total crosses a threshold. That's the whole operation.

Think about what that operation actually does. It adds everything up and asks: is the total above the threshold? If yes, fire. If no, don't. That's one question, asked once. Everything on one side of that question is "yes." Everything on the other side is "no."

That's what people mean when they say a neuron can only draw one line. Not a line you can see, a single dividing question that splits the world in two.

For simple problems, that's enough. Is this number bigger than five? Is this email short or long? One rule can answer those.

But most real problems aren't like that. Think about sorting photographs into cats and dogs. The difference between a cat and a dog isn't one thing you can measure and compare. It's a combination of things: the shape of the ears, the structure of the face, the proportions of the body. None of those features, on its own, gives you the answer. The answer lives in how they combine.

One neuron, one rule, one line. It can't find that.

Citations(2)↓
  1. 1. deeplearningbook.org
  2. 2. doi.org
Citations(2)↓
  1. 1. deeplearningbook.org
  2. 2. doi.org