Loading slide
Loading contents...
Nobody taught you the rule you just used. You have read enough sentences about things and containers to know that a thing too big will not go inside, and a container too small will not hold what is put in it.
The model faces that task with none of your experience. It has the tokens, and what it learned during training.
And it starts from a real disadvantage. The stored embedding for it is the same list of numbers in both sentences, so the answer cannot be sitting in that list. It has to come from the tokens around it.
In the first sentence, trophy and big should influence what it becomes. In the second, suitcase and small should influence it. The other tokens matter less.
That is attention's job: decide which available tokens should contribute more, and which should contribute less.
Some models receive the whole finished sentence at once, so their attention can use tokens before and after it. That is what made the last token available to settle the question.
# did you know?
The trophy and suitcase pair comes from a family of language puzzles called Winograd schemas.
Each puzzle changes one small clue so that a word such as it points somewhere different. The point is to make shallow word matching fail.