Loading slide

The Representation Problem

  1. 01The Representation Problem
  2. 02The gap between pixels and words
  3. 03How a lesson spreads
  4. 04Why the numbers have to mean something
  5. 05Go on, try it
  6. 06So add a direction
  7. 07One-hot encoding
  8. 08The question that breaks it
  9. 09What we actually need
  10. 10You already do this
  11. 11Meaning is relational
  12. 12Reinforce your understanding
  13. 13Question: Why labels fail
  14. 14Question: Meaning from company
  15. 15Question: What good numbers do
  16. 16Quiz: answer
  17. 17What to carry out of this chapter
  18. 18Want to go deeper?
6 / 17
BackNext

So add a direction

While you were fighting that line, you may have felt the fix yourself. Stuck trying to keep "king" near "man" while "queen" and "woman" kept getting in the way, the natural thought is: why can't "man" just sit behind "king," off the line?

That small thought is the entire answer. The instant you say "behind," you have asked for a second direction, and a plain line doesn't have one, it only runs left and right. Give yourself an up-and-down as well as a left-and-right, and the trap springs open.

Lay the four words out on a flat square instead of a line:

  • "king" top-left, "queen" top-right
  • "man" bottom-left, "woman" bottom-right

Now read it off. The two royals share the top row, so they're close. The two ordinary people share the bottom row, close. The two males share the left column, close. The two females share the right column, close. Every single relationship that was impossible on the line now holds without a fight. The loop that couldn't be flattened onto a line sits perfectly happily as a square. What changed? Each word now has two directions to be near things in: one direction carries royalty, the other carries gender, and a word can be near one thing along the first while being near a different thing along the second.

Two directions handled two kinds of relationship. But words relate to each other in far more than two ways. Think about everything "dog" is close to, and why it's close to each: to "puppy" by age, to "wolf" by wildness, to "cat" by being a pet, to "bark" by sound, to "loyal" by temperament. Each of those is a different kind of closeness, a different direction, its own "behind." To keep them all true at the same time, two directions won't do. You need dozens. Real systems use hundreds.

So here is the real shape of the answer. A word cannot be a single number on a line, or even a dot on a flat page. It has to be a point sitting in a space with hundreds of directions at once, near some words along one direction, near completely different words along another. Nobody can picture a space like that, and that is genuinely fine, the coming slides give us honest ways to think about it without picturing it. But you have already earned the key fact: a word's meaning pulls in too many directions to ever fit on a single line.

Which leaves a very practical question. We can't possibly hand-place every word in a space of hundreds of directions; that would take forever and we'd never agree on where things go. So the first real attempt didn't even try to capture meaning. It just tried to give each word numbers that, at the very least, never lie. That attempt is where we go next.

Citations(2)↓
  1. 1. arxiv.org
  2. 2. jalammar.github.io
Citations(2)↓
  1. 1. arxiv.org
  2. 2. jalammar.github.io