The reply streams back

As the sampler selects new token numbers, the program running Rufus keeps the growing reply.

The tokenizer now works in the opposite direction. It turns the generated token numbers back into readable text.

The program does not need to wait for the complete reply. It can send small pieces of text across the internet while the reply is still being generated. This is called streaming.

Tom’s chat app displays the pieces as they arrive:

“Hi Tom. I’m Rufus, your friendly assistant.”