The model can't read

Here is something that surprises most people.

A language model has never read a single word. Not one. It cannot. Everything a computer touches has to become numbers first. That has been true since the very first module, and language is no exception.

The last chapter was about what kind of numbers could carry meaning. But hiding underneath that was a more basic question, and we skipped over it. Before you can hand out numbers at all, you have to decide what gets one. What are the pieces? Whole words? Single letters? Something else?

Neither obvious answer works. Give every letter its own number and words dissolve: "dog" becomes d, o, g, three pieces that have nothing to do with barking. Give every whole word its own number and the list never ends: every name, every typo, every word someone invents next year would need a new entry.

The answer sits in between, and it has a name: tokenization.