The loop that was being scaled

Before we talk about making it bigger, be clear about what it is.

Training a language model is the same guess-and-adjust loop you already know, pointed at text. Show it a stretch of writing, let it guess what comes next, tell it the real answer, nudge the weights a hair toward it. Then again.

The loop is not the interesting part. The number of times it runs is.

Billions of passes, over trillions of words, across thousands of processors working at once, for weeks or months without stopping. Each single pass changes the weights by an amount too small to notice. Nothing about one pass is clever. The whole of the result comes from doing an unremarkable thing an unreasonable number of times.

Here is the strange part. Nobody ever tells the model what grammar is. Nobody hands it a list of facts. Those things arrive as a side effect of getting good at one plain task, guessing the next piece of text.

So the question the field started asking around 2017 was simple to say and hard to imagine. What happens if you just make this loop much, much bigger?

The loop that was being scaled

Before we talk about making it bigger, be clear about what it is.

The loop is not the interesting part. The number of times it runs is.

So the question the field started asking around 2017 was simple to say and hard to imagine. What happens if you just make this loop much, much bigger?