Meaning falls out of prediction

Here is how that idea becomes a method. It is the same loop that trained the image networks, pointed at language instead of pictures.

In 2003, Yoshua Bengio and colleagues showed a neural language model that learned a useful number pattern for each word while learning to predict text. The prediction task and the representation were trained together.

Take millions of real sentences. Hide a word. Show the neighbours. Make the model guess the missing word.

At the start it guesses badly. Each wrong guess gets nudged a little, just like before.

To get good at this game, the model has to notice which words keep the same company. If dog and puppy keep turning up among the same neighbours, then whenever one of them is hidden, guessing the other is a safe bet. So the model learns to treat them as nearly interchangeable.

The model was never asked to learn meaning. It was only asked to predict neighbours. But to predict neighbours well, it has to place words that share company close together. The moment it does that, the map builds itself.

Meaning falls out of prediction as a side effect.