What the model actually sees

The symbols were a stand-in. What a token actually carries is a number.

Every piece in the vocabulary has one, fixed when the vocabulary was built. In one common tokenizer, never is 37593, trust is 7095, and a is 264. So the pirate sentence arrives like this:

37593, 7095, 264, 55066, 449, 264, 2472

That row is the whole input. 264 appears twice because a appeared twice, and on the surface that repetition is the only structure there is.

Now, that number is not trying to mean anything. 55066 does not say pirate is large, or nautical, or anything at all. It is an address: the 55,066th entry in a list, assigned by the order the vocabulary happened to be built in.

Which is what the last chapter warned about. An ID alone is never enough. But the tokenizer was never the part meant to fix that.

Its job was to decide what gets a number at all, and that question is now answered: not words, not letters, but tokens. A fixed list of tens of thousands of pieces, every one of them with an address.

What happens to that address is that it gets looked up. Every token in the vocabulary has an entry in a table, and the entry is not one number but hundreds, learned from use and positioned so that pieces used alike sit near each other.

That list of numbers is the embedding, the thing the last chapter named and could not yet build.

Before that, though, the vocabulary has one more consequence, and it lands on people rather than on models. Which pieces made it into the list was decided by whoever built it, and not everyone was in the room.