Loading slide
Loading contents...
There are thousands of languages and millions of words, and more invented every year. How does a fixed list of a few tens of thousands of pieces cover all of that?
Go back to the codebook from the very first module: 65 means A, 66 means B. Eight switches, 256 patterns, and that group of eight has a name: a byte. One byte per character works until you try to write anything but English. There are far more than 256 characters in the world, and a single byte runs out.
The fix is the move this chapter has already made twice. When one byte is not enough, use several. a still takes one, but ä takes two, 字 takes three, and 🍄 takes four. The character is not stored in a byte. It is spelled across as many as it needs.
So the tokenizer keeps all 256 patterns in its vocabulary, and that alone can spell anything anyone has ever written: a language it never saw, a name nobody has invented yet, a symbol from a dead script. Worst case it goes one byte at a time, slowly, and it never gets stuck.
Which changes what the rest of the list is for. Reach was settled with those 256 entries. Every entry after them buys speed: the turns up so constantly that it was worth storing whole rather than spelling out every time.
And there is the answer to why Swedish costs more. It was never shut out. English filled the pile, so English fragments won the stored entries, and Swedish falls back toward spelling things out. It gets the same reach. It just does not get the shortcuts.