More than one kind of relevance

One more refinement, and it is a natural one once you ask the obvious question: relevant how?

When you read "it," you are not tracking just one kind of connection. Part of you is asking "which noun does this stand for?" Another part is tracking grammar, what is the subject, what is the verb. Another might be following the mood of the sentence, or which words rhyme, or which belong to the same topic. "Relevant" is not a single thing. There are many different kinds of relationship running at once.

A single round of attention, one want-ad per word, can only chase one kind of relevance at a time. So the machine simply runs several rounds in parallel, each with its own set of want-ads and chest-cards. Picture several readers looking at the same sentence side by side: one reader hunting for what each pronoun refers to, another tracking grammar, another following the topic. Each builds its own set of weights, its own blend, and then their findings are combined.

These parallel readers are called . Nobody assigns the jobs. Nobody tells head one to do grammar and head two to do reference. Just like AlexNet working out edges and shapes for itself in the going-deeper module, the heads discover their own specialities through training, simply because dividing the labour makes them better at the task.

So the full picture is not one web of connections over the sentence, but several, laid over each other, each tracking a different thread of meaning. Hold that image, because it is the raw material of the machine in the next chapter. Stack these multi-headed attention layers up, drop the old single-file reading entirely, and you have built a transformer.