Loading slide
So a network learns about one number and spreads that lesson to its neighbours. With pixels, the neighbours really are alike, so the spreading helps. Now let's hand it our word-numbers and watch the very same habit wreck everything.
Recall how we numbered the dictionary: "dog" landed on 3, "dollar" on 4, "puppy" out around 6,000. To the network these are just positions on its street of numbers, and it has no idea what any of the words mean. It only knows where they sit.
Now the network learns something true about "dog," say, that the word "barked" often comes right after it. Good. But remember the spreading. The moment it learns that about number 3, it leans over and assumes number 4 behaves similarly, because next door means alike. Number 4 is "dollar." So the network starts expecting "dollar barked," confidently, for no reason other than that we happened to seat "dollar" beside "dog." The lesson spread to a complete stranger.
Meanwhile the word that should have caught the lesson, "puppy," is sitting 6,000 doors down the street. It is so far away that none of the spreading reaches it. So the network never carries what it learned about dogs over to puppies, even though those are the two words that most belong together. The help goes to the stranger and skips the relative. Every useful lesson leaks to the wrong words and never reaches the right ones.
This is the heart of it. The network is not broken; it is doing exactly the smart thing that made it work on images. We simply handed it a street where the house numbers are nonsense, where next-door neighbours are unrelated and true relatives live at opposite ends of town. On a map like that, "spread the lesson to your neighbours" sends every lesson to the wrong address.
So the problem with words was never "they are hard to turn into numbers." You can do that in a second; we just did. The problem is making the closeness between those numbers tell the truth, so that "dog" sits near "puppy" and far from "dollar," and the network's spreading finally lands on the right doors.
And here you might reasonably object: fine, then let me just pick better numbers. Put "dog" at 2 and "puppy" at 3 so they're neighbours, and shove "dollar" far away. Now the closeness is honest, isn't it? Problem solved. Hold that thought, because it is exactly the right thing to try, and watching it fall apart is the whole reason everything that follows had to be invented.