Scale and Emergence

The last module handed us a finished machine: the transformer. We know what it is made of and how it reads a whole sentence at once. This module asks the question that machine kept pointing at. What happens when you make it very, very large?

The honest answer is that something happens that nobody fully expected.

There is a kind of change that creeps along, small and steady, and then crosses a line and turns into something else. Heat water degree by degree and for a long while you just get warmer water. Keep going and at one exact temperature it stops being water at all and becomes steam. Same heat, same pot, but past a threshold the rules change.

Something like that happened as these models grew. More parts, more training text, more raw computing power. For a while they simply got better at the task they were built for: guessing the next word. Then, past a certain size, they started doing things no one had built into them. The ability to translate, to do arithmetic, to follow a recipe of instructions. Nobody put those abilities in. They showed up.

That is the word for this module: emergence. A new ability that appears from sheer scale, not from any single part. It is not magic, and we will take it apart carefully. But it surprised the people building these systems, and it is where the large-model story really begins.