Vocabularies, byte pair encoding, the seams in unbelievable. If the details have blurred, one thing is worth keeping.

Text does not reach a model as text. It arrives cut into pieces from a fixed list, each piece swapped for its number, and the cuts fall where letter-counting put them rather than where the meaning is.

Almost everything odd about these systems at the input end follows from that. Letters sealed inside a piece cannot be counted. A word the vocabulary never learned costs more to say. The same sentence is cheaper in the language the list was built from.

None of it was designed to work out that way. It is what you get when the only question ever asked is which letters turn up together.

And the number a piece ends up with means nothing at all. It is an address. What lives at that address, and how anything gets there, is the last thing this module has to explain.