The idea is older than you think

One more thing about that comparison, because the history is genuinely surprising.

In 1948, published a paper mostly about telephone lines, and buried in it is the ancestor of everything in this chapter.

He generated English text by statistics. By hand.

His method was comically physical. Open a book, pick a word. Flip to another page, read until you find that word, write down whatever follows it. Then find that word somewhere else and take its follower. Then do the whole thing again using two previous words instead of one, and again using three.

The samples beside this slide are that experiment, run over the text of Pride and Prejudice. With no context you get word salad. With one word, pairs begin to hold. With two, grammar appears from nowhere. With three it is close to something a person might have written badly.

Shannon could not have known where that observation led. He had no computer worth the name, no digital text, and no way to look back more than a few words.

The insight that more context yields better prediction is that old. What took another seventy years was building a machine that could act on it at scale, which is a decent summary of this entire journey.

Step through the four samples with that in mind. Nothing in them understands anything. Each was built by counting which words followed which, then drawing from those counts, and the only difference between the worst and the best is how far back the counting looked. A transformer looks back across thousands of tokens rather than three, and weighs every one of them against every other, but it is answering the same question Shannon was asking with a book and a pencil.