Reading the request, writing the reply

A request to draft an invitation might contain the event notes, earlier messages, and instructions about tone.

The model first processes the input. Much of this work can happen across the supplied tokens together.

Then it generates the reply in sequence. Each next-token choice depends on the input and the output so far. A longer reply requires more generation work.

Software can save intermediate attention calculations and reuse them. This is a . It does not update the trained weights.

Long input takes work to process and hold. Long output adds generation steps. Both use computing resources.