A space of meaning

The first chapter left us with a wish list. A word should be a point in a space with hundreds of directions, near its relatives, far from its strangers, with closeness that tells the truth. This chapter is that space, actually built.

Each token from the last chapter maps to a . Where one-hot encoding wasted ten thousand slots saying nothing, these are a few hundred numbers that all say something.

Read them as coordinates. A point on a paper map needs two, an x and a y. A word needs hundreds, placing it somewhere in a .

Nobody can picture hundreds of directions, and nobody needs to. Near and far still work exactly the way they do on a map.

What matters is where words land. Words used in similar ways end up close together, not because anyone arranged them, but because, as "pripple" showed us, words with similar meanings keep similar company.

The map on this slide has no headings on it. Wander it anyway and the groupings announce themselves: a cluster of animals, a cluster of feelings, the capitals huddled together. Nobody sorted them. They ended up neighbours by being used alike, and distance finally means something.

It is a sketch, though, with hundreds of directions flattened onto a page that has only two. Hover Germany and its neighbours light up as France, England, Japan, Italy. Not Berlin, even though Germany and Berlin plainly belong together.

You have seen this before. It is the bracelet: lay a loop of four beads out in a row and one bond always stretches, whichever bead you cut between. Here the two bonds are Germany-to-France and Germany-to-Berlin, and a flat page has no direction left to give the second one.

In the real space there is no such squeeze. Germany sits near the other countries along one direction and near Berlin along another, and hundreds of directions leave room for both.

So the neighbourhoods on this map are real, and a missing neighbour is usually a casualty of the flattening rather than a fact about the words.

A space of meaning

Each token from the last chapter maps to a . Where one-hot encoding wasted ten thousand slots saying nothing, these are a few hundred numbers that all say something.

Read them as coordinates. A point on a paper map needs two, an x and a y. A word needs hundreds, placing it somewhere in a .

Nobody can picture hundreds of directions, and nobody needs to. Near and far still work exactly the way they do on a map.

In the real space there is no such squeeze. Germany sits near the other countries along one direction and near Berlin along another, and hundreds of directions leave room for both.

So the neighbourhoods on this map are real, and a missing neighbour is usually a casualty of the flattening rather than a fact about the words.