Loading slide

Embeddings

  1. 01Embeddings
  2. 02Memory card
  3. 03Hold this one idea
  4. 04A space of meaning
  5. 05Don't panic about hundreds of directions
  6. 06Word2Vec and the arithmetic of meaning
  7. 07Let that land
  8. 08Memory card
  9. 09The same word, different meanings
  10. 10Reinforce your understanding
  11. 11Question: Geometry of meaning
  12. 12Question: What embeddings can't capture
  13. 13Quiz: answer
  14. 14What you now know
  15. 15Meaning became geometry
  16. 16Want to go deeper?
7 / 15
BackNext

Stop for a second, because what you just saw is genuinely one of the strangest results in this whole field, and it is easy to skim past without feeling how odd it is.

Nobody taught the model that kings and queens are royalty, or that "man" and "woman" differ by gender. Nobody wrote those rules down anywhere. The model did one mindless thing, billions of times: cover a word, guess it from its neighbours, adjust. That is all.

And out the other end fell arithmetic on meaning. King, take away man, add woman, and you arrive at queen. The idea of "changing the gender" turned into an actual direction you can walk in the space, the same direction every time, for every pair. A concept became a move.

Here is the friendly way to hold it. Because similar words had to sit near each other, and this had to be true for every word at once, the only way to satisfy all those constraints was to line the words up so that shared relationships pointed the same way. Consistency was forced on the space from the outside, by the sheer weight of fitting every word in at once. The arithmetic is just that consistency, made visible.

You do not need to fully resolve how astonishing this is. Just carry the fact: relationships between words became directions you can add and subtract. That single fact is a large part of why machines can suddenly do so much with language.

Citations