This module followed the move from symbols to learned positions.

• In 2003, neural language models learned word representations while predicting text.
• In 2013, Word2Vec made large word maps practical and exposed useful geometric relationships.
• In 2016, subword tokenization helped neural translation work with rare and unseen words.
• By 2018, contextual systems such as ELMo gave the same word different representations in different sentences.

A token ID is only an address. The stored embedding at that address is a learned list of numbers. Training places tokens used in similar ways near one another.

What geometry cannot do on its own is read a sentence. Every position is settled before the sentence arrives, so bank sets off from the same spot whether the river or the money is meant. The words around it have to lean in and adjust it, which means working out which of them matter.

The next module builds the mechanism that lets surrounding words make that adjustment.