Loading slide
Here is a question that should be nagging at you. The world has thousands of languages and millions upon millions of distinct words. So how on earth does a vocabulary of only tens of thousands of tokens cover all of it? Surely we would need millions of tokens, not a hundred thousand.
The resolution is one of the neatest ideas in this whole module, and it rests on a single fact about how computers store text.
Every character on Earth, an English letter, a Chinese character, an emoji, a symbol from a script you have never seen, is ultimately stored as one or more bytes. A byte is just a number from 0 to 255, and here is the crucial part: there are only 256 possible bytes in existence, ever. That is a hard ceiling. Common characters take one byte; rarer ones are stored as a short sequence of several bytes glued together.
You might wonder how a computer knows whether three bytes in a row are three separate characters or one bundled-up character. It is the same trick as the file signature from the very first module: the opening bits of each byte act as a little label, announcing "I stand alone" or "I am the start of a bundle, read the next ones with me." As that JPEG slide put it, the bits never change, only the agreement about how to read them does. We do not need the details here. The point that matters is simpler: it is all built from the same tiny set of 256.
So watch what happens when the tokenizer simply includes those 256 bytes in its vocabulary as tokens. The moment it does, it can already spell anything. Any word, any language, any emoji, because all of it is made of those same 256 bytes. Total coverage of every language ever written, bought with just 256 tokens. Nothing is ever truly missing. Worst case, the tokenizer spells a rare word out one byte at a time, slowly, but it never gets stuck.
Which reframes the whole vocabulary. Coverage was never the expensive part. It is already free.
So what are the other ninety-thousand-odd tokens for? Speed, and nothing else. They are shortcuts. The tokenizer notices that " the" appears constantly, so rather than spell it out byte by byte every single time, it buys one whole token for " the". It does this for the most common fragments it sees. These shortcuts let you say frequent things in fewer pieces. They do not let you say anything new, the bytes already covered everything.
And now your earlier puzzle clicks shut. This is exactly why some languages cost more. English filled most of the training pile, so English fragments earned most of the precious shortcut-tokens, and English packs tight. A language that barely appeared earned few shortcuts, so it falls back toward slow, byte-by-byte spelling. It is never excluded. It just does not get the discounts.
256 tokens buy the ability to write everything. The other hundred thousand just buy speed on the things that come up often.