Try it yourself

This is the real thing: a tokenizer used by real language models, running in your browser. Type anything.

Try your own name. Try "strawberry" and watch it come apart into pieces that have nothing to do with how the word sounds. Invent a word on the spot: the tokenizer has never seen it, but it copes, spelling it out of smaller pieces it does know. Then try a sentence in another language and count how many more tokens it takes.

Keep one eye on the numbers under each piece. The colored chunks are drawn for you. The model gets only the row of IDs.