Text becomes tokens

Rufus does not receive the request exactly as Tom sees it.

Because Rufus is , the tokenizer also runs on the provider’s servers. It sits between the incoming request and Rufus.

The tokenizer breaks the text into pieces called tokens. One possible split might look like this:

User : Hello , I’m Tom .

Each token is matched with a number. Rufus receives those numbers in the same order.

Different LLMs may split the sentence differently, but the purpose is the same: turn text into numbers Rufus can work with.