Loading slide
Loading contents...
The fix is clever in how ordinary it is.
If attention cannot tell where a word sits, then write it on the word before it goes in. Each position gets its own distinct pattern of numbers, added on top of the word's meaning, before anything enters the tower. A
Think of name tags at a long table. Everyone wears a tag showing both their name and their seat number. Now you can shuffle the guests into any heap you like and still reconstruct who sat where. Nothing about the heap has changed. The information came along inside the tags.
So dog in seat two is no longer the same list of numbers as dog in seat five. When attention compares words, it is comparing things that carry their positions with them, and the dog bit the man no longer looks like its reverse.
Transformers represent position in several ways. Some add learned position patterns rather than fixed ones. Others adjust the attention comparison itself, using the distance between two words rather than their absolute seats. The implementation varies. The job stays the same: give attention usable information about order.
# did you know?
The original transformer made its position stamps from numerical waves that repeat at different speeds.
One wave changes quickly and another slowly. Read together, their values give each seat a distinctive pattern.