Loading slide

Scale and Emergence

  1. 01Scale and Emergence
  2. 02Memory card
  3. 03The loop that was being scaled
  4. 04What happens when you just make it bigger
  5. 05The improvement had a shape
  6. 06Hold this contrast
  7. 07Abilities that appear from nowhere
  8. 08Take a breath
  9. 09The GPT-3 moment
  10. 10A raw model is not yet helpful
  11. 11Teaching it to answer
  12. 12Letting humans steer it
  13. 13The side effect: confident and wrong
  14. 14Reinforce your understanding
  15. 15Question: What emergence means
  16. 16Question: Making the model helpful
  17. 17Question: Confident and wrong
  18. 18Quiz: answer
  19. 19Bigger changed what was possible
  20. 20Want to go deeper?
BackNext chapter

Look at the ground this chapter covered. We started with one humble loop, guess the next word, the same guess-and-adjust move from the learning module. We made it enormous. And out the other side came two surprises stacked on each other: a smooth, predictable climb in skill, and on top of it, brand-new abilities switching on at sizes no one could call in advance. That is emergence, the wetness that only appears once you have enough water.

Then we watched a raw text-continuer get shaped into the assistant you actually use. Hand-written good answers to imitate, then human preferences to lean toward. And we saw the price tucked into that polish: a model trained to sound confident sounds confident even when it should not, which is where hallucination comes from.

That leaves one oddly basic question. Moment to moment, as a model writes a reply, what is it actually doing?

The answer is simpler than almost anyone expects, and stranger for being so simple. That is the next chapter.

Citations