Loading slide
Loading contents...
Before a language model touches a sentence, a
A token is not the same as a word. It might be a whole word: dog is usually one token. It might be a piece of a word, so a longer or rarer word arrives in two or three parts. It might be a single punctuation mark, a space, or a number.
Where do the splits come from? From a fixed vocabulary of common fragments learned before the language model is trained. One common method is
In 2016, researchers adapted this kind of repeated merging to neural translation. It helped a model handle rare and unseen words without keeping one vocabulary entry for every whole word. Modern tokenizers use several related methods, but the practical trade remains the same.
The broad pattern is simple: frequent fragments are more likely to become one token. Rarer text is assembled from smaller pieces.
# did you know?
Byte pair encoding was introduced in 1994 as a way to compress computer files, not as a way to process language.
Researchers later reused its habit of repeatedly joining common neighbours to build text pieces for neural translation.