Three lists from every token

To choose the weights, an attention layer makes three new lists of numbers from every token.

Each list has one job.

The first list is used to look for useful tokens. It is called the query.

The second list is used to show what a token can match with. It is called the key.

The third list holds the information that the token can add to the mix. It is called the value.

These lists are patterns of numbers, not written descriptions. Their numbers can capture useful clues, such as whether a token behaves like a noun or verb, where it appears, and what it means. The model learns these clues during training.

For it, the query asks what matters, the keys help find a match, and the values provide the information that gets mixed in.