Loading slide
One more oddity, because it surprises almost everyone. The token for "cat" in the middle of a sentence is " cat", with the space stuck on the front. A bare "cat" with no space is a different token, with a different ID.
Why glue the space on? Because the space is a real character, as real as any letter, and it has to be stored somewhere. Rather than spend a separate token on every space, the tokenizer tucks each space onto the front of the word that follows it. And since almost every word in normal writing has a space in front of it, " cat" is the version you see nearly all the time. The spaceless "cat" is kept for the few places a word has nothing before it: the very start of a sentence, or right after a bracket, as in "(cat)".
Does that mean every word needs two tokens? It is a fair worry, but no. Keeping both spellings costs two slots in the dictionary, paid once when the vocabulary is built. It does not cost you two tokens each time you write the word. Every spot in your text still uses exactly one token; the tokenizer just reaches for whichever spelling fits and ignores the other. So this slightly pads the size of the dictionary, never the length of your sentence. And only common words earn the second spelling at all; rarer ones keep one and spell the odd case out of smaller pieces.
There is a quiet lesson here. To you, "cat" and " cat" are obviously the same word. To the model they start as two unrelated numbers, and it has to learn, from seeing them used in the same company, that they mean the same thing. Which is exactly the idea from the last chapter: meaning is not handed over, it is worked out from the company a token keeps.