Loading slide

Loading contents...

[███████████░░░░░░░][██████████████████░░░░░░░░░░]12 / 19
<back>next

The space in front of the word

You already made this happen. strawberry gave three pieces; the same word with a space in front gave one.

A space is a real character and has to go somewhere, so the tokenizer glues it onto the front of the word that follows.

Now ask where the bare word strawberry, with nothing in front of it, actually appears in the that this model has trained on. Almost nowhere: at the start of a sentence, and little else. And how many sentences begin with the word strawberry? Almost none. Nearly every time anyone writes it, it sits a few words in, with a space in front.

So that is the version the counting saw over and over, and (space)strawberry earned an entry of its own. The bare word never came up enough to earn one, which is why it falls back to str, aw, berry.

So one version arrives as a single ID and the other as three, and nothing about those numbers says they are the same word. The model works that out from use, because the two turn up in the same company constantly.

What looks like one word to you can be several entries, depending on what sits beside it.

# citations(2)↓
  1. [1]huggingface.co
  2. [2]github.com