Word2Vec and the arithmetic of meaning

In 2013, researchers at Google published Word2Vec, a practical way to learn word vectors from nearby words in large collections of text. One version predicts a word from its neighbours. Another uses a word to predict nearby words. In both, repeated prediction gradually shapes a map.

The learned space did more than place related words nearby. Some relationships turned up as a repeated step: not just how far apart two words sit, but which way you go to get from one to the other.

The famous example is king - man + woman. Take the step that carries man to woman. Go back to king, take that same step again, and you arrive near queen. The walk on this slide does exactly that.

Near, not on. The step lands in the right neighbourhood rather than dead on the word, it works for some pairs and not others, and a different model gives a different result.

No one entered a table saying which words were royal. The geometry came from patterns in the training text. That also means the map can inherit the text's gaps and stereotypes.

Word2Vec and the arithmetic of meaning

The learned space did more than place related words nearby. Some relationships turned up as a repeated step: not just how far apart two words sit, but which way you go to get from one to the other.

Near, not on. The step lands in the right neighbourhood rather than dead on the word, it works for some pairs and not others, and a different model gives a different result.

No one entered a table saying which words were royal. The geometry came from patterns in the training text. That also means the map can inherit the text's gaps and stereotypes.