Loading slide

Tokens

  1. 01Tokens
  2. 02Memory card
  3. 03The model can't read
  4. 04Not words. Tokens.
  5. 05The emoji trick
  6. 06Try it yourself
  7. 07How many r's in strawberry?
  8. 08What the model actually sees
  9. 09Some languages cost more
  10. 10Why "cat" and " cat" differ
  11. 11Why not millions of tokens?
  12. 12So why not just use bytes?
  13. 13Context windows are measured in tokens
  14. 14Reinforce your understanding
  15. 15Question: Why not just use words?
  16. 16Question: Weird splits
  17. 17Quiz: answer
  18. 18What to carry out of this chapter
  19. 19Want to go deeper?
9 / 18
BackNext

Some languages cost more

Here is a consequence that sounds unfair, because it is. Write the same sentence in two languages, and the model can charge you very different amounts of tokens for it.

Take a plain greeting. In English, "Hello, how are you?" lands in about 6 tokens. Translate it into a language the tokenizer saw less of during training, and the very same meaning can take two or three times as many. The thought is identical. The token bill is not.

Why does this happen? Go back to how the vocabulary was built. The tokenizer learned its fixed set of pieces by scanning a huge pile of text, and that pile was mostly English. So English words got the most dedicated tokens: "hello" is common, so it earned a whole token of its own. A word from a language that barely appeared in the pile never earned that prize. When it shows up, the tokenizer has no single piece for it, so it falls back to spelling it out of much smaller scraps, sometimes letter by letter. More scraps means more tokens for the same word.

It gets worse for writing systems that do not use the English alphabet at all. A single Chinese character or an emoji can be split into several tokens on its own, because at the very bottom the tokenizer falls back to raw bytes, the tiny numbers a computer uses to store any character. One character you would write in a single stroke can cost three or four tokens.

Why does this matter? Tokens are the unit everything is counted in, so this quietly taxes everyone who does not write in English. It shows up in three ways.

The first is money. Using a model is usually billed per token, so the same email costs a Hindi or Burmese speaker more than an English speaker, for no extra value.

The second is speed. A model produces text one token at a time, so more tokens means a slower reply.

The third is memory. A model can only hold so many tokens in mind at once, a limit we will name properly in a moment. If your language burns tokens faster, you hit that ceiling sooner. Less of your document fits, and the model "forgets" earlier parts of the conversation quicker.

None of this was decided on purpose. Nobody set out to charge some languages more. It simply falls out of building the vocabulary from text that was mostly English, and then measuring everything in the pieces that vocabulary happens to contain.

Citations(2)↓
  1. 1. arxiv.org
  2. 2. huggingface.co
Citations(2)↓
  1. 1. arxiv.org
  2. 2. huggingface.co