Loading slide

Embeddings

  1. 01Embeddings
  2. 02Memory card
  3. 03Hold this one idea
  4. 04A space of meaning
  5. 05Don't panic about hundreds of directions
  6. 06Word2Vec and the arithmetic of meaning
  7. 07Let that land
  8. 08Memory card
  9. 09The same word, different meanings
  10. 10Reinforce your understanding
  11. 11Question: Geometry of meaning
  12. 12Question: What embeddings can't capture
  13. 13Quiz: answer
  14. 14What you now know
  15. 15Meaning became geometry
  16. 16Want to go deeper?
9 / 15
BackNext

The same word, different meanings

Word2Vec had one stubborn limitation: each word got a single, fixed position. But words shift meaning depending on where they sit.

"Bank" by a river is not the "bank" you deposit money in. A "bright" light and a "bright" student are related but not the same. In a fixed embedding, both senses of a word collapse onto one point, an average smeared across every context the word ever appeared in. The model can't tell which "bank" you meant.

The fix is called . Instead of one position per word, the position is recalculated for each sentence, shaped by the words around it. "Bank" drifts toward one neighborhood when "river" is nearby and a different one when "account" is.

Every large language model works this way. A word isn't just dropped into the space of meaning, it's placed according to its whole sentence, weighed against every other word around it. But doing that, letting every word look at every other word and adjust, demands a specific machine built for the job. That machine is the transformer, and it is the entire next module.

Citations(2)↓
  1. 1. arxiv.org
  2. 2. arxiv.org
Citations(2)↓
  1. 1. arxiv.org
  2. 2. arxiv.org