Loading slide
Loading contents...
This is the real thing: a tokenizer used by real language models, running in your browser. Type anything.
Try your own name. Invent a word on the spot: the tokenizer has never seen it, but it copes, spelling it out of smaller pieces it does know. Then try a sentence in another language and count how many more tokens it takes.
Then try this. Type strawberry and watch it break into three pieces. Now clear the box, press the space bar once, and type the word again, exactly as it would appear inside a sentence.
One token. The same ten letters, and the space in front changed everything about how they arrive. Keep that in mind, because it explains a failure these models became famous for.
# did you know?
Type unbelievable here and you get different pieces than the ones shown a few slides back. Neither is wrong. This playground runs one tokenizer, that example came from another, and every model brings its own.
A tool called Tiktokenizer lets you switch between them. Common words usually land the same way whichever you pick. Awkward ones are where they part company.