Three ways to wire a network

It helps to see what the transformer is an alternative to.

Every design in this course is a neural network. What separates them is how information gets from one part of the input to another.

• Recurrent. Pass a running note along, one step at a time.
• Convolutional. Slide a small window across the input, looking at a little of it at a time. Mostly used for images.
• Transformer. Let everything look at everything, all at once.

Underneath, all three do the same arithmetic on the same kind of thing: multiply some numbers, add them up, repeat. There is no special language machinery inside a transformer that a convolutional network lacks. Both are piles of numbers with the same operations running over them.

What differs is the wiring, and the wiring decides what a network is able to learn.

Sliding a small window builds in an assumption, that things near each other belong together. A network built that way cannot take in a whole sentence at once, however long you train it, because nothing in its structure reaches that far. Letting everything reach everything builds in the opposite assumption, and pays for it in work.

Same arithmetic. Different reach.

Back to the tower, then, and to the part of a floor that has been left out. Because a floor does not only pay attention.