Loading slide

Embeddings

  1. 01Embeddings
  2. 02Memory card
  3. 03Hold this one idea
  4. 04A space of meaning
  5. 05Don't panic about hundreds of directions
  6. 06Word2Vec and the arithmetic of meaning
  7. 07Let that land
  8. 08Memory card
  9. 09The same word, different meanings
  10. 10Reinforce your understanding
  11. 11Question: Geometry of meaning
  12. 12Question: What embeddings can't capture
  13. 13Quiz: answer
  14. 14What you now know
  15. 15Meaning became geometry
  16. 16Want to go deeper?

Module complete

Next: 07: Paying Attention

BackStart module 07

Words are points in space. Similar words sit nearby. Relationships are directions. Nobody designed any of it; the structure emerged from a model playing the guessing game on enough text.

One refinement remains. A fixed point per word blurs words like "bank" that mean several things, so the position has to be recomputed from the surrounding sentence, every single time. Doing that, letting every word weigh every other word before settling into place, takes a machine built specifically for the job.

That machine is the transformer, and it is the next module.

Citations