Loading slide
Before we talk about making things bigger, we should be precise about what was being made bigger.
Training a language model is a loop, and it is the same loop from the learning-from-data module, just pointed at text. The model sees a stretch of writing, a few hundred or a few thousand chunks of it, and guesses what comes next. Then it is shown the real answer. If it guessed wrong, its weights, the millions of little numbers inside, get nudged a tiny step toward the right answer. Then the next stretch of text. Then the next.
That is the whole loop: see text, guess the next piece, check the answer, nudge the weights. Read it, and it should feel familiar. It is guess and adjust, the exact move from when we first watched a single neuron learn.
This loop runs billions of times, over trillions of words, across thousands of processors at once. Each pass shifts the weights by a hair. Stacked up over enough passes, those hairs become a model whose guesses match the real patterns in the text.
Here is the part worth sitting with. Nobody ever tells the model what grammar is. Nobody hands it a list of facts. Those things arrive as a side effect of getting good at one plain task: guess the next piece of text. So the question the field started asking around 2017 was simple to say and hard to imagine. What happens if you just make this loop much, much bigger?