How many r's in strawberry?

For a while this was the joke everyone told about language models. Ask one how many times the letter r appears in strawberry, and it would answer two.

A machine that could write working software, confidently wrong about a question a child answers by pointing at the letters. People took it as proof that the whole thing was hollow.

The reason is on the previous slides. Type that question and the word arrives as one token. Not s-t-r-a-w-b-e-r-r-y. One marker standing for the whole word, the way 🍄 stood in for a word a moment ago.

That is the one-token version you just made happen, by putting a space in front of the word.

So the question being asked is stranger than it looks. Count the r's in 🍄. The letters are not in there to be counted. They were sealed inside the token before the model saw anything, and no amount of care at that point can open it back up.

It is not that the model failed to look properly. It never had the letters.

A model can still learn spelling, because text is full of people talking about how words are spelled, and it picks that up the way it picks up anything else. That is knowing about the letters rather than seeing them, which is why the answers were unreliable rather than always wrong. Given a tool that counts characters directly, it gets the answer right every time.