The emoji trick

A token's ID is a number, but nothing about the idea needs it to be. The only thing the model requires is that each distinct piece of text gets its own marker, and that the marker is the same every time that piece shows up. Numbers are just what computers have to hand.

So swap them for something easier to look at. Here is a sentence with every word replaced by a symbol picked at random:

never trust a pirate with a map

🌀 🎺 🍄 🪝 🔮 🍄 🗝️

These words are all short and common, so each one happens to be a single token. That is why the symbols match the words one for one here. Longer words would break into several, and nothing else about this would change.

Look only at the symbols and you are seeing roughly what the model gets. Not words. A row of markers, each standing in for a chunk of text. Nothing in them says these were ever language, and nothing says one of them is about pirates.

Notice that a appears twice and gets the same symbol both times. The same fragment becomes the same token wherever it turns up, in this sentence or any other, forever. The model's entire world is built from a fixed set of these markers, tens of thousands of them, and nothing else.

What it does with them, how it learns which tend to follow which, is a later story. For now, the words you see and the markers the model sees are not the same thing.