Loading slide
Loading contents...
Take a long word like unbelievable. You would break it where the meaning breaks: the bit that reverses it, the bit about belief, the bit that makes it a quality.
A real tokenizer cuts it as un, bel, ievable. Another cuts it somewhere else again. Which one hardly matters. The pieces are fragments of the word, and no tokenizer is trying to make them line up with the meaning.
It could not try. A tokenizer is a counting program. It has no notion of meaning to look for, no idea that believ is worth keeping together or that ievable is nonsense, because it does not know what words are for.
The word you would like is one solid brick. What arrives is three smaller bricks with the seams in the wrong places.
That comes from how the pieces were chosen. Byte pair encoding starts with single letters and finds the pair that occurs most often across a huge pile of text. Suppose t and h win. Every th gets glued into one piece, and it counts again. Now maybe th and e are the most common neighbours, so the becomes a piece. Repeat tens of thousands of times and you have a vocabulary.
Nowhere in that process does anything ask what a word means. There is nothing to ask with. The only question ever put is which letters keep turning up together.
That is how bel earned a place. Look at where else it turns up: believe, belong, belly, beloved, and rebel all arrive carrying that same piece. Those five words have nothing to do with one another. A belly is not a belief. But the three letters appear side by side often enough to be worth storing once, and so they were.