The matching

To update it, the model compares its query with the key from every available token.

Each comparison produces a number called a score. A stronger match gets a higher score. A weaker match gets a lower score.

The model then turns all those scores into weights. The weights add up to 1.

A large weight lets more of that token's value enter the mix. A small weight lets less enter.

The key is used for the match. The value is what gets added. Keeping those jobs separate lets the model learn one number pattern for finding a useful token and another for the information that token should contribute.

So the path is:

1. Compare one query with every key.
2. Turn the scores into weights.
3. Use the weights to mix the values.

The matching

To update it, the model compares its query with the key from every available token.

Each comparison produces a number called a score. A stronger match gets a higher score. A weaker match gets a lower score.

The model then turns all those scores into weights. The weights add up to 1.

A large weight lets more of that token's value enter the mix. A small weight lets less enter.

So the path is:

1. Compare one query with every key.
2. Turn the scores into weights.
3. Use the weights to mix the values.