The Problem with Sequences

The last module ended on a promise. To handle a word like "bank," a model has to look at the whole sentence around it and weigh every other word before deciding what that word means here. We said that took a machine built specifically for the job, and we even named it: the transformer. This module is that machine, built up from nothing.

But you cannot feel why the transformer was such a leap until you feel the wall it broke through. So this first chapter is not about the transformer at all. It is about how machines handled language before it, and the single problem that older approach could never shake.

The problem is distance. To understand a sentence, you are forever connecting words that sit far apart. An "it" near the end points back to a noun near the start. A conclusion leans on a condition set out sentences ago. The older approach read text the way you might read through a keyhole: one word at a time, left to right, with everything before it slowly fading as new words arrived. The further back something was, the harder it was to hold on to.

That was not a bug someone forgot to fix. It was the shape of the whole approach, and it put a hard ceiling on what these systems could do.