Loading slide

A Network That Learns

  1. 01A Network That Learns
  2. 02Stacking neurons
  3. 03Quick recall
  4. 04What stacking does
  5. 05Hidden layers
  6. 06The problem of credit
  7. 07Backpropagation
  8. 08Sharing the blame
  9. 09The search problem
  10. 10Fog on a hill
  11. 11Gradient descent
  12. 12Just to be clear
  13. 13What the hidden layer learns
  14. 14What the hidden layer discovers
  15. 15Did it actually learn?
  16. 16Generalization
  17. 17Limits in the 1990s
  18. 18Reinforce your understanding
  19. 19Question: Backpropagation
  20. 20Question: What the hidden layer learns
  21. 21Question: Memorizing vs. generalizing
  22. 22Quiz: answer
  23. 23The method existed. What was missing was scale.
  24. 24Want to go deeper?
5 / 23
BackNext

Hidden layers

And just like riding a bike, nobody programs in the exact angles to lean at or the forces to apply. The network finds those things through repetition and correction, the same way your brain did. The weights adjust, layer by layer, until something works.

This kind of structure has a name: a neural network with hidden layers. The middle layers are called hidden not because they're secret, but because you can't see them from the outside. The inputs go in. The output comes out. Everything in between is invisible unless you go looking.

Did you know?

This invisibility is a real problem, even for the people who build these networks. When a neural network makes a wrong call, it is often genuinely unclear which layers or weights were responsible. Researchers call this the black box problem.
There is an entire field dedicated to opening the box: mechanistic interpretability. Its goal is to figure out what individual neurons and layers are actually doing, what they have learned to detect, and why. It is one of the hardest open problems in AI research.

The architecture was the easy part. Training it was a different problem entirely.

Citations(2)↓
  1. 1. deeplearningbook.org
  2. 2. doi.org
Citations(2)↓
  1. 1. deeplearningbook.org
  2. 2. doi.org