Attention is all you need

In 2017, a team of eight researchers, most of them at Google, published a paper called "Attention Is All You Need." They were not trying to build a chatbot. They were trying to translate between languages, and the machine they described had two halves: one to read a sentence, one to write it out in another language.

To feel what that title was claiming, remember where attention came from. It arrived in 2014 as an addition. The recurrent network was still the machine: still reading word by word, still carrying its running note from one word to the next. Attention was bolted on beside it, a way for the system to reach back to an earlier word instead of trusting the note to have kept it. A helpful extra part on a machine that already worked.

The 2017 paper proposed removing the recurrent network and keeping the extra part.

No reading in order. No running note. No waiting for the previous word to finish. Every word looks at every available word directly, all at once, and that is the only way information moves between them.

That is what all means in the title. Not a boast about how good attention is, but a statement about what was left: nothing else. The supporting act was asked to carry the show alone.

It worked better than the thing it replaced.