A tower of floors

What they built is called a transformer, and it is easiest to meet as a picture before it is a definition.

Picture a tower of floors.

At the bottom, the tokens come in, each one a position in the space of meaning where related words sit near each other. Then the first floor: a full round of attention, every token reaching out, weighing every other available token, blending in what matters.

Out the top of that floor comes a new version of each token, now coloured by its context. it may carry a little trophy. bank may lean toward river or money, depending on its neighbours.

That result passes to the next floor, and the whole thing happens again. Then again. One round of attention barely changes a word. Dozens of rounds, each working on what the last one produced, is where the depth comes from.

The name is a fair clue to all this. A transformer transforms: each word arrives as one list of numbers and leaves as a different one, changed to fit the sentence it is sitting in, and then it happens again on the floor above. What the name leaves out is what does the changing. That part is the words consulting each other.

So attention is one step, inside one floor. The transformer is the whole tower.