Wiring that changes with the sentence

That second step, the desk work, is a plain neural network. The kind you already know. A few layers of neurons, each connection carrying a weight that training settled, and every word put through the same one.

Attention is the half that does not look like that.

In an ordinary network the wiring is fixed. Neuron three connects to neuron seven with a weight of, say, 0.4, and that 0.4 is the same whatever you feed in. Training set it once, and it stays. That is what a trained network is: a very large number of settled connections.

How much it draws from trophy is not settled anywhere. Nothing in the model stores it. It is worked out from the two words at the moment they meet, and a different sentence produces a different amount.

So the connections between words are not part of the machine. They are calculated fresh for every sentence, and they are gone once the sentence is done.

That is the unusual thing about attention. Training decides how words compare. The sentence decides what comes of the comparison.