What Attention Means

Understanding often means knowing what to ignore.

When you read a sentence, you don't weigh every word equally. Some words matter more than others for understanding what's happening right now. You track them, return to them, let them shape how you read what comes next.

Attention, in the technical sense, is a mechanism that lets a machine do something similar. For any given piece of text, it computes how much each other piece of text should influence the meaning of this one. Not a fixed rule. A learned, context-sensitive weighting.

The effect is that distant words can have direct influence on each other, without anything having to travel through all the words in between. Long-range relationships become tractable in a way they weren't before.

That single idea turned out to be more powerful than almost anyone expected.