Loading slide

The Representation Problem

  1. 01The Representation Problem
  2. 02The gap between pixels and words
  3. 03How a lesson spreads
  4. 04Why the numbers have to mean something
  5. 05Go on, try it
  6. 06So add a direction
  7. 07One-hot encoding
  8. 08The question that breaks it
  9. 09What we actually need
  10. 10You already do this
  11. 11Meaning is relational
  12. 12Reinforce your understanding
  13. 13Question: Why labels fail
  14. 14Question: Meaning from company
  15. 15Question: What good numbers do
  16. 16Quiz: answer
  17. 17What to carry out of this chapter
  18. 18Want to go deeper?
5 / 17
BackNext

Go on, try it

Let's take that objection seriously, because it really is the right instinct. We get to choose the numbers, so let's choose them by hand and put every word next to its true relatives. And let's not be fussy about it: "close" doesn't mean glued together. A word only has to land nearer its relatives than its strangers. "Cat" can sit a few doors down from "wolf" and that's fine, as long as both are a long way from "vegetable." We have plenty of slack.

So we start placing words, and at first it goes well. Then it falls apart, and it falls apart for a reason that has nothing to do with how carefully we choose. The trouble is the shape we are placing them on. A single number puts every word somewhere on one long line, like our street of house numbers, and a line forces you to commit to one single order. Once you fix that order, it has to keep every relationship true at the same time. Watch it break with just four words.

Take "king," "queen," "man," and "woman." Look at how they relate:

  • "king" and "queen" belong together (royalty)
  • "man" and "woman" belong together (ordinary people)
  • "king" and "man" belong together (both male)
  • "queen" and "woman" belong together (both female)

Every one of the four needs to sit close to two of the others. Now try to put them in a single line so all four of those closenesses hold at once.

Say you line them up in this order: king, queen, woman, man. Check it. King beside queen, good. Queen beside woman, good. Woman beside man, good.

But now look at the two ends of the line. "King" and "man" have been flung to opposite ends, as far apart as four things can be, and those two are supposed to be close, because they are both male.

Broken.

Try another order and you just move the break somewhere else.

Picture the four words as four beads on a closed bracelet, strung in the order king, queen, woman, man, and then back round to king. On the bracelet every bead touches its two neighbours and nobody is left out, because a loop has no ends.

Now take the bracelet off and lay the beads out in a straight row. The moment you do, two of the beads have to sit at the two ends of the row. A bead on the end has only one neighbour, with nothing on its far side. So a pair that was happily touching on the bracelet gets pulled apart.

It does not matter which order you pick. A row always has two lonely ends and a loop never does, so flattening the loop into a row always snaps exactly one of the four bonds.

Notice what this means: the failure has nothing to do with picking better numbers. We could fiddle with the exact values forever and never fix it. The line itself is too cramped. It offers only one direction to spread out in, and meaning, as those four words just showed us, pulls in more than one direction at once.

Citations(2)↓
  1. 1. arxiv.org
  2. 2. ibm.com
Citations(2)↓
  1. 1. arxiv.org
  2. 2. ibm.com