Loading slide

The Representation Problem

  1. 01The Representation Problem
  2. 02The gap between pixels and words
  3. 03How a lesson spreads
  4. 04Why the numbers have to mean something
  5. 05Go on, try it
  6. 06So add a direction
  7. 07One-hot encoding
  8. 08The question that breaks it
  9. 09What we actually need
  10. 10You already do this
  11. 11Meaning is relational
  12. 12Reinforce your understanding
  13. 13Question: Why labels fail
  14. 14Question: Meaning from company
  15. 15Question: What good numbers do
  16. 16Quiz: answer
  17. 17What to carry out of this chapter
  18. 18Want to go deeper?
10 / 17
BackNext

You already do this

We just asked for a map where a word's place is decided by its meaning. But that only pushes the problem back a step. Where would the meaning itself come from? Nobody can type in what 10,000 words "mean." So before the next slide names the trick, do it yourself, on one sentence.

Read this:

The waiter brought us a bottle of wibble, and it paired well with the fish.

You have never seen the word "wibble" before. It isn't a word. And yet you already have a fairly good guess about what it is. Some kind of wine, probably, or at least a drink you have with dinner.

Stop and notice how you knew that. You did not look "wibble" up anywhere; there was nothing to look up. You read the company it was keeping. A waiter brought it. It came in a bottle. It paired with fish. Each neighbouring word ruled some things out and ruled others in, until "drink you have with a meal" was almost the only thing left standing.

That is the idea, and you ran it without being taught. The meaning of a word is not stored inside the word. It is smeared across all the places the word tends to show up. See enough of a word's company and the meaning falls out on its own.

It is the same thing you do learning a foreign language. Nobody hands you a definition for every word; you hear "she poured the leche into the coffee," then "a glass of cold leche," then "the baby drinks leche," and "milk" starts to take shape from the company the word keeps. That is also why being surrounded by a language teaches you faster than flashcards: a flashcard shows you the word alone, stripped of its company, while real sentences show it in the neighbourhood where its meaning actually lives.

Now imagine doing that not with one sentence, but with millions of them, for every word at once. Words that keep similar company, "dog" and "puppy," forever surrounded by bark, leash, pet, fur, would slowly be pulled toward the same neighbourhood on the map. Not because anyone defined them. Because they kept the same company.

The next slide gives this idea its proper name, and the moment it was first written down.

Citations(2)↓
  1. 1. cs.brown.edu
  2. 2. ibm.com
Citations(2)↓
  1. 1. cs.brown.edu
  2. 2. ibm.com