Loading slide

Scale and Emergence

  1. 01Scale and Emergence
  2. 02Memory card
  3. 03The loop that was being scaled
  4. 04What happens when you just make it bigger
  5. 05The improvement had a shape
  6. 06Hold this contrast
  7. 07Abilities that appear from nowhere
  8. 08Take a breath
  9. 09The GPT-3 moment
  10. 10A raw model is not yet helpful
  11. 11Teaching it to answer
  12. 12Letting humans steer it
  13. 13The side effect: confident and wrong
  14. 14Reinforce your understanding
  15. 15Question: What emergence means
  16. 16Question: Making the model helpful
  17. 17Question: Confident and wrong
  18. 18Quiz: answer
  19. 19Bigger changed what was possible
  20. 20Want to go deeper?
6 / 19
BackNext

The next slide is the strangest idea in this module, so let us set up the one contrast that makes it click. It is a contrast between two things scale does, and they are not the same.

The first you just saw: the smooth, predictable kind of improvement. Make the model bigger and it gets steadily, reliably better at what it already does. That is the scaling-law curve, a clean line you could draw in advance. No surprises. More size, a bit more skill, smoothly.

The next slide is about the other kind, and it is nothing like smooth. Sometimes, past a certain size, a model can suddenly do something it simply could not do at all before. Not "a little better." Off, then on. A brand-new skill, appearing at a size nobody could predict.

So as you read on, keep both in one hand: scale makes old skills smoothly better, and scale also makes entirely new skills suddenly appear. The first was expected. The second was the shock. That gap between the smooth curve and the sudden jump is the whole point of what comes next.

Citations