Loading slide

The Neuron Idea

  1. 01The Neuron Idea
  2. 02A familiar idea
  3. 03Same era, different bet
  4. 04A neuron, at its simplest
  5. 05Adding things up
  6. 06The first artificial neuron
  7. 07Strength, not just presence
  8. 08The signal itself has a strength
  9. 09Multiply, then add
  10. 10Just a recipe
  11. 11What a weight really is
  12. 12Where the numbers come from
  13. 13Back to the real thing
  14. 14The Perceptron
  15. 15How it learned
  16. 16What training actually is
  17. 17Why this was different
  18. 18What Rosenblatt claimed
  19. 19The shape of hype
  20. 20Twelve years of real work
  21. 21The verdict
  22. 22What the proof didn't say
  23. 23What it could do
  24. 24What it could not do
  25. 25The reason
  26. 26The limit was real
  27. 27Before we go further
  28. 28Memory card
  29. 29What if we added another?
  30. 30Many dumb things
  31. 31Layers
  32. 32More Layers
  33. 33The whole thing
  34. 34A network of neurons
  35. 35A small group kept going
  36. 36Reinforce your understanding
  37. 37Question: What is learning?
  38. 38Question: The limit
  39. 39Question: Training a network
  40. 40Quiz: answer
  41. 41The idea didn't die. It waited.
  42. 42Want to go deeper?
25 / 41
BackNext

The reason

All the Perceptron does is multiply and add. Each input gets multiplied by a weight. All of it gets added together. That's the entire operation.

The problem is that each weight is fixed. One number, decided during training, applied every single time that input appears.

Go back to the train ticket. What weight should it get? If the murder happened somewhere else, the ticket means nothing, the weight should be zero. If the murder happened on that train, the ticket is everything, the weight should be huge. But those are two different answers. The Perceptron has to pick one and stick with it.

It can't say "it depends." It has no way to make one input's weight change based on what another input is doing. Every input is scored alone, in isolation, every time.

Think of a recipe where every ingredient just adds its own flavour independently. More salt makes it saltier. More pepper makes it spicier. Simple, predictable.

But some flavours only work together. Fish sauce on its own tastes awful. Lime juice on its own is just sour. Combined, in the right dish, they become something neither one is alone. No amount of adjusting individual weights will find that.

That's not a gap in the training. It's a gap in the design.

Citations(2)↓
  1. 1. deeplearningbook.org
  2. 2. mitpress.mit.edu