Loading slide

Embeddings

  1. 01Embeddings
  2. 02Memory card
  3. 03Hold this one idea
  4. 04A space of meaning
  5. 05Don't panic about hundreds of directions
  6. 06Word2Vec and the arithmetic of meaning
  7. 07Let that land
  8. 08Memory card
  9. 09The same word, different meanings
  10. 10Reinforce your understanding
  11. 11Question: Geometry of meaning
  12. 12Question: What embeddings can't capture
  13. 13Quiz: answer
  14. 14What you now know
  15. 15Meaning became geometry
  16. 16Want to go deeper?
4 / 15
BackNext

A space of meaning

The first chapter left us with a wish list. A word should be a point in a space with hundreds of directions, near its relatives, far from its strangers, with closeness that tells the truth. This chapter is that space, actually built.

Each token from the last chapter maps to a . Where one-hot encoding wasted ten thousand slots saying nothing, these are a few hundred numbers that all say something.

Read them as coordinates. A point on a paper map needs two, an x and a y. A word needs hundreds, placing it somewhere in a .

Nobody can picture hundreds of directions, and nobody needs to. Near and far still work exactly the way they do on a map.

What matters is where words land. Words used in similar ways end up close together, not because anyone arranged them, but because, as "wibble" showed us, words with similar meanings keep similar company.

The map on this slide is a sketch, with the hundreds of directions squashed down to two so we can look at them. But neighbourhoods like these are exactly what forms. Wander it: animals in one region, feelings in another, capitals huddled together. Distance finally means something.

Citations(2)↓
  1. 1. arxiv.org
  2. 2. jalammar.github.io
Citations(2)↓
  1. 1. arxiv.org
  2. 2. jalammar.github.io