A private slot

Start with the smallest job. Forget relatedness for a moment. Just give the computer a way to tell one word from another.

A single ID already does that. dog as 3 and dollar as 4 are different, so the computer can tell them apart. The problem is that 3 and 4 also sit next to each other, and a network treats neighbouring numbers as similar.

So do not use one number. Give every word its own private slot in a long row. For a 10,000-word vocabulary that is 10,000 slots. dog turns one of them on, say slot 2,742, and leaves the rest off. puppy turns on a different slot. No two words share one.

Written as numbers, an on slot is a 1 and an off slot is a 0. So dog becomes a row of ten thousand numbers that are all 0 except for a single 1.

Now there is no 3 sitting next to 4. Any two different words are equally far apart. The computer can still tell them apart. It no longer claims that neighbours on a number line belong together.

People call that . It is not obsolete either. Today's models still begin there: a word arrives as a private slot, and the first thing the model does is turn that slot into a richer list of numbers.

A private slot

Start with the smallest job. Forget relatedness for a moment. Just give the computer a way to tell one word from another.

Written as numbers, an on slot is a 1 and an off slot is a 0. So dog becomes a row of ten thousand numbers that are all 0 except for a single 1.

Now there is no 3 sitting next to 4. Any two different words are equally far apart. The computer can still tell them apart. It no longer claims that neighbours on a number line belong together.

People call that . It is not obsolete either. Today's models still begin there: a word arrives as a private slot, and the first thing the model does is turn that slot into a richer list of numbers.