Loading slide

The Representation Problem

  1. 01The Representation Problem
  2. 02The gap between pixels and words
  3. 03How a lesson spreads
  4. 04Why the numbers have to mean something
  5. 05Go on, try it
  6. 06So add a direction
  7. 07One-hot encoding
  8. 08The question that breaks it
  9. 09What we actually need
  10. 10You already do this
  11. 11Meaning is relational
  12. 12Reinforce your understanding
  13. 13Question: Why labels fail
  14. 14Question: Meaning from company
  15. 15Question: What good numbers do
  16. 16Quiz: answer
  17. 17What to carry out of this chapter
  18. 18Want to go deeper?
1 / 17
BackNext

Module VI

The Meaning of Words

Chapter I

The Representation Problem

In this chapter

  • Why images were easy and language was hardthe gap between pixels and words
  • One-hot encodingwhat it is, what it misses, and why that matters
  • Meaning is relationalwhy a word's meaning comes from the company it keeps
  • What a good representation would actually need to capture

Before a machine can learn from something, that something has to become numbers.

The last module ended on exactly this wall. Deep learning had conquered images, because an image is already numbers: a grid of pixel brightnesses, one value per dot of color. Feed those in and learning can begin. Then we said language was next, and language was harder. This module is why.

Words are not numbers. You can assign them numbers, of course, but an arbitrary label carries nothing about meaning. The number you hand to "dog" tells the network nothing about how "dog" relates to "puppy" or "animal" or "bite." The label is not the thing.

This is the representation problem: not just translating words into a form machines can process, but translating them in a way that preserves something of what they mean. Without that, a machine can process language without having any grip on it.

How you solve that problem turns out to matter enormously.

Citations