Loading slide

Scale and Emergence

  1. 01Scale and Emergence
  2. 02Memory card
  3. 03The loop that was being scaled
  4. 04What happens when you just make it bigger
  5. 05The improvement had a shape
  6. 06Hold this contrast
  7. 07Abilities that appear from nowhere
  8. 08Take a breath
  9. 09The GPT-3 moment
  10. 10A raw model is not yet helpful
  11. 11Teaching it to answer
  12. 12Letting humans steer it
  13. 13The side effect: confident and wrong
  14. 14Reinforce your understanding
  15. 15Question: What emergence means
  16. 16Question: Making the model helpful
  17. 17Question: Confident and wrong
  18. 18Quiz: answer
  19. 19Bigger changed what was possible
  20. 20Want to go deeper?
8 / 19
BackNext

Take a breath

Let us pin down the two ideas so far, because the rest of the module leans on them.

The first is the smooth curve. Make a model bigger and it gets steadily, predictably better at the thing it was already doing: guessing the next word. That part surprised no one once they saw it. It is the loaf that grows with the ingredients.

The second is the jump. On top of that steady improvement, brand-new abilities switch on at sizes nobody could call ahead of time. Translation, arithmetic, following instructions. Off, then on, like wetness appearing once you have enough water. That part surprised everyone, including the people who built the models.

Hold both at once and you have the strange engine of this whole era. A simple loop, guess the next word, scaled up beyond anything imagined, producing first predictable gains and then unpredictable new powers.

Now we can look at the moment the wider world first felt this, when one model got large enough that it stopped feeling like a tool and started feeling like a surprise.

Citations