Loading slide

The Transformer

  1. 01The Transformer
  2. 02Memory card
  3. 03Attention is all you need
  4. 04A stamp for position
  5. 05Take a breath
  6. 06Two ways to use it
  7. 07Train once, adapt anywhere
  8. 08Why it took over everything
  9. 09Reinforce your understanding
  10. 10Question: Why transformers took over
  11. 11Question: Pretraining and finetuning
  12. 12Question: Where word order went
  13. 13Quiz: answer
  14. 14The architecture everything is built on
  15. 15Want to go deeper?
1 / 14
BackNext

Module VII

Paying Attention

Chapter III

The Transformer

In this chapter

  • "Attention is all you need"the claim, and the tower of attention layers behind it
  • A stamp for positionhow the transformer knows word order without reading in order
  • Two ways to use itthe fill-in-the-blank reader and the keep-writing generator
  • Train once, adapt anywherethe pattern that has shaped AI ever since

Now and then a new design does not just beat what came before. It makes everything before it look like a long detour.

The last chapter built one part: attention, a way for every word to reach out, weigh every other word, and blend in the ones that matter. This chapter is the machine built entirely out of that part. Its claim, made in a paper with a title that doubled as a dare, was startling in its plainness. You do not need the old single-file reading. You do not need anything clever bolted on. Stack attention, and that is enough.

That machine is the transformer. Give it nothing but a mountain of text and one absurdly simple thing to practise, and it soaks up grammar, facts, and the texture of language on its own. Make it bigger, and it starts doing things nobody trained it to do. Almost every AI system you have heard of is, underneath, one of these.

This chapter is how the part becomes the machine, plus the two small pieces attention needed before it could stand on its own.

Citations