Naming the thing

These machines have a common name, and it has not appeared once in this course so far. They are called large language models. The phrase is worth taking apart, because you will meet it constantly.

Start with language model. It means a system that can say how likely each possible next word is. That is the whole definition, and it says nothing about how the system works inside.

The category is old. The first language models were built in the 1970s and 1980s and were not neural networks at all. They counted. Go through a large pile of text, and every time dog appears, record the word that came after it. Do that for every word. Later, asked what follows dog, answer barked, because barked was recorded most often.

No weights. No training. Just counting, and then looking up what was counted.

Now large. Read plainly, it only claims a difference of size, and that is not what happened.

The distance between those counting machines and GPT-3 is not a matter of counting harder. The insides were thrown out and rebuilt more than once. Neural networks replaced counting in the 2000s, so the machine learned patterns instead of tallying occurrences. Recurrent networks came next. Then the transformer. Only then was the result made enormous.

So the phrase is a poor description of its own subject. Language model names the job, predicting the next word, and that job has not changed since the 1970s. Large is the only word left to cover four decades of rebuilt machinery, and all it actually reports is the size of the last version.