Loading slide
Loading contents...
Take the word (space)strawberry again. In English that is one token, a single entry in the list. In Swedish it is jordgubbe, and the same fruit arrives in five pieces: j, ord, g, ub, be.
Not one of those is a word. The tokenizer is spelling the fruit out of scraps, and it happens right through the language: a whole Swedish paragraph takes around half as many tokens again as the same paragraph in English.
The reason is the counting from earlier. Whichever runs of letters turned up most often in the pile of text earned their own tokens, and that pile was overwhelmingly English. English words earned whole entries. Swedish ones mostly did not, so they get assembled from fragments instead.
Nothing is broken. Swedish still goes in and Swedish still comes out. It just takes more pieces to say the same thing, and every one of those pieces costs something.
It is not a blanket rule, either. tack, Swedish for thank you, is one token where thank you is two. But across a whole document, the language the tokenizer was built on wins comfortably.