Loading slide
Before a language model touches a sentence, a
A token is not the same as a word. It might be a whole word ("dog" is usually one token). It might be a piece of a word ("unbelievable" splits into "un", "belie", "vable"). It might be a single punctuation mark, a space, or a number.
Where do the splits come from? From a fixed vocabulary of common fragments, learned once from a huge sample of text by a method called
Why bother with fragments instead of whole words? Think of how LEGO works.
You could imagine a toy kit with one custom moulded piece for every object a child might ever build: a special piece shaped like a castle, another shaped like a spaceship, another like a giraffe. The factory could never keep up. The moment a child wanted something new, there would be no piece for it.
LEGO does the opposite. It ships a modest set of standard bricks, and out of those same bricks you can build the castle, the spaceship, the giraffe, and a million things the designers never pictured.
A few reusable pieces beat an infinite catalogue of finished ones.
Whole-word tokens are the custom-moulded approach. You would need a separate piece for every inflection ("run", "runs", "running", "ran"), every name, every typo, every technical term, and every word someone invents next year, a list that never stops growing and still misses whatever comes next. Fragments are the LEGO bricks. With a fixed set of common pieces you can build almost any word by snapping smaller known parts together, including words the vocabulary has never seen.
How big does that set of bricks need to be? For many language models, the answer is tens of thousands of pieces. Far smaller than the millions of possible words, but enough to spell out almost anything.
The exact number does not matter and will drift over time. What matters is the trade: a small, fixed box of reusable parts instead of an endless catalogue of finished words.