Loading slide
We just saw that a single number is far too cramped: one direction, when meaning needs hundreds. The first real attempt to escape that, years ago, went all the way to the opposite extreme. It gave each word a huge pile of numbers instead of one. It did fix the cramping, but it fixed it so bluntly that it threw out the very thing we wanted. It is worth seeing exactly how, because its failure points straight at what a good answer has to do.
The scheme is called
Written as numbers, an on switch is a 1 and an off switch is a 0. So each word becomes a row of ten thousand numbers that are all 0 except for a single 1. "Dog" is the row with the 1 sitting in slot 2,742 and 0 everywhere else. "Puppy" is the row with its 1 in slot 6,015. Every word gets its own private switch, and no two words ever share one.
Now see what this was designed to buy us. The thing that ruined the single-number street was accidental closeness: "dog" and "dollar" ending up next-door neighbours for no reason, so lessons spread between them. One-hot makes that impossible. No two words sit next to each other, because no two words share any switch at all. Each word is off in its own slot, touching nothing. There are no accidental neighbours, because, as we're about to see, there are no neighbours of any kind.
And that is the catch hiding in plain sight. At first glance one-hot looks like it finally solved the problem: no lies, no false closeness, every word clearly distinct. Hold that thought for one slide. We're going to ask a single simple question about these switch-rows, and the whole thing comes apart.