Loading slide

The Translator at the gate

A Translator hurried forward inside the temple.

She broke the written question into small pieces and replaced each piece with a number. Then she placed the numbers before the Oracle.

The Oracle could finally begin her work.

Behind the story+
The Translator stands for the tokenizer. A tokenizer follows a fixed method for turning text into token numbers. It changes the form of the message but does not decide what the words mean. The language model produces the answer.
# citations(1)↓
  1. [1]huggingface.co

Loading contents...

[████░░░░░░░░░░░░░░][██████░░░░░░░░░░░░░░░░░░░░░░]4 / 18
<back>next