Rufus scores token numbers

Rufus uses the complete input to score every possible next token number.

During training, Rufus saw many examples of how tokens appear together in real text. He learned patterns from those examples.

He now uses those learned patterns to score how well each possible next token fits everything that came before. Each number corresponds to a token, so we can translate a few possibilities back into readable pieces:

For example:

2418 → Hi might receive a high score.

9906 → Hello might receive a lower score.

6712 → Thanks might receive an even lower score.

Rufus has not selected a word or token. He has produced scores attached to token numbers.