Abilities that appear from nowhere

Smaller models couldn't do arithmetic. Then, past a certain scale, they could.

Smaller models couldn't translate languages they hadn't been fine-tuned on. Then, past a certain scale, they could.

Smaller models couldn't follow multi-step instructions, write structured code, or explain their own reasoning. Then, past a certain scale, they could — and nobody had trained them explicitly to do any of these things.

Researchers called these emergent abilities. The word comes from physics and biology: properties that arise from a system's complexity that aren't present in any individual component. A single neuron doesn't produce consciousness. A single water molecule isn't wet. Something new appears at scale.

The unsettling part: these abilities appeared at unpredictable thresholds. You couldn't always tell in advance what a model would be able to do until it crossed a certain size. The capability wasn't in the training data as an explicit target. It emerged from the combination of scale and a model flexible enough to represent it.

<!-- TODO: a chart showing capability near-zero at small scale then jumping sharply at a threshold would illustrate emergence intuitively -->