The Representation Problem

Before a machine can learn from something, that something has to become numbers.

For images, this is straightforward. A photograph is already a grid of values, one number per pixel per color channel. Feed those numbers into a network and learning can begin.

Language is different. Words are not numbers. You can assign them numbers, but arbitrary labels carry no information about meaning. The number you give to "dog" tells the network nothing about how "dog" relates to "puppy" or "animal" or "bite." The label is not the thing.

This is the representation problem: not just translating words into a form machines can process, but translating them in a way that preserves something of what they mean. Without that, a machine can process language without having any grip on it.

How you solve that problem turns out to matter enormously.