Loading slide
You may have seen the joke. Someone asks a powerful language model a question a five-year-old can answer, "how many r's are in the word strawberry?", and the model confidently says two. The internet had a field day. If these things are so clever, how can they fail at counting letters?
But look at what you just watched in the playground. "Strawberry" did not arrive as the letters s-t-r-a-w-b-e-r-r-y. It arrived as a handful of chunks, something like "str", "aw", "berry", each one a single token with one ID number standing in for the whole chunk. The model is handed those three IDs and nothing else. The individual letters were never passed along; from the model's side they are not separate things sitting inside the chunk, they simply are not there.
So asking the model to count the r's is like asking someone to count the bricks in a photo of a finished wall. The wall is right there, but the seams between the bricks have been plastered over. It is not that the model is bad at counting. It is that, from where it sits, there is nothing to count. The thing you are asking about, the row of letters, is not the thing it was ever given.
This is the first time tokenization stops being a technical detail and starts explaining something you can see. Hold onto it: a surprising number of "look how dumb the AI is" moments turn out to be tokenization showing through, not stupidity.