All at once, not one at a time

Attention also changed how much of the work could happen at once.

Look back at the RNN. It was stuck reading in order: word one, then word two, then word three, each step waiting on the one before, because the running summary had to be updated in sequence. You could not start word one hundred until you had trudged through the ninety-nine before it. The work was a single-file line, and a line can only move as fast as one worker.

Attention has no such waiting. Scoring it against trophy needs nothing from suitcase or because. During one use of attention, the model can calculate many token-to-token matches together instead of crawling along a line.

And you have met exactly the hardware that loves this kind of work. Remember the orchestra: the , thousands of small workers all computing side by side, brilliant whenever a task breaks into many independent pieces. The RNN handed that orchestra a single-file line and wasted it. Attention hands it thousands of independent little matches, exactly what it was built to chew through all at once.

Attention solved two problems at once. Relevant tokens could connect directly, and the model could calculate many matches at the same time.